2024-03-28T15:30:14.259 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T15:30:14.259 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} --name mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --archive /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315 --owner mchangir -- /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/orig.config.yaml 2024-03-28T15:30:14.316 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T15:30:14.406 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} email: mchangir@redhat.com first_in_suite: false job_id: '7625315' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount dry_run: true install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3645 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 132/256 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi023.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKe2XaK9HWq33BrKGOTSxvkkDwiSln13+HoUlnMOxnPKisYU0HCrPUT39s6k1s7W7dSOC8TsF7ahA39Ny69wxA= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOKIwvfVbhEEto+ufiP/WeBcqdpEAiC5Npdu0Pn+eVJafT8N5Qx11/74qcv6FomniomKYvzxZPdRZHMd+MPyIWI= smithi120.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyLah9Ww/6g3VLLFTwgmiScvB4FsQ+BcrTIkRx01rkcfEwkPFuIU9wGCETnGqY4NyaqNN/SjlV8/J1YiZ2+Z+Y= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsync-tester.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-27_03:45:23 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T15:30:14.406 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-28T15:30:14.407 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-28T15:30:14.407 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T15:30:14.408 INFO:teuthology.task.internal:Checking packages... 2024-03-28T15:30:14.432 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-28T15:30:14.432 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T15:30:14.432 INFO:teuthology.packaging:ref: None 2024-03-28T15:30:14.432 INFO:teuthology.packaging:tag: None 2024-03-28T15:30:14.432 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-28T15:30:14.432 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:30:14.432 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-28T15:30:14.695 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-28T15:30:14.696 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T15:30:14.703 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T15:30:14.703 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T15:30:14.712 INFO:teuthology.task.internal:Saving configuration 2024-03-28T15:30:14.731 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T15:30:14.736 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T15:30:14.757 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.929993', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKe2XaK9HWq33BrKGOTSxvkkDwiSln13+HoUlnMOxnPKisYU0HCrPUT39s6k1s7W7dSOC8TsF7ahA39Ny69wxA='} 2024-03-28T15:30:14.779 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.933697', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOKIwvfVbhEEto+ufiP/WeBcqdpEAiC5Npdu0Pn+eVJafT8N5Qx11/74qcv6FomniomKYvzxZPdRZHMd+MPyIWI='} 2024-03-28T15:30:14.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.931971', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyLah9Ww/6g3VLLFTwgmiScvB4FsQ+BcrTIkRx01rkcfEwkPFuIU9wGCETnGqY4NyaqNN/SjlV8/J1YiZ2+Z+Y='} 2024-03-28T15:30:14.801 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T15:30:14.808 INFO:teuthology.task.internal:roles: ubuntu@smithi023.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-28T15:30:14.808 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-28T15:30:14.808 INFO:teuthology.task.internal:roles: ubuntu@smithi120.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-28T15:30:14.808 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T15:30:14.901 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdff8b3a0d0>, signals=[15]) 2024-03-28T15:30:14.902 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T15:30:14.908 INFO:teuthology.task.internal:Opening connections... 2024-03-28T15:30:14.908 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:30:14.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:30:15.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-28T15:30:15.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:30:15.102 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi120.front.sepia.ceph.com 2024-03-28T15:30:15.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:30:15.191 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T15:30:15.198 DEBUG:teuthology.orchestra.run.smithi023:> uname -m 2024-03-28T15:30:15.216 INFO:teuthology.orchestra.run.smithi023.stdout:x86_64 2024-03-28T15:30:15.216 DEBUG:teuthology.orchestra.run.smithi023:> cat /etc/os-release 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:NAME="CentOS Stream" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION="9" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:ID="centos" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:VERSION_ID="9" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:ANSI_COLOR="0;31" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:30:15.273 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:30:15.274 INFO:teuthology.orchestra.run.smithi023.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:30:15.274 INFO:teuthology.lock.ops:Updating smithi023.front.sepia.ceph.com on lock server 2024-03-28T15:30:15.290 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-28T15:30:15.308 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-28T15:30:15.308 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:30:15.365 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:30:15.366 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:30:15.366 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-28T15:30:15.390 DEBUG:teuthology.orchestra.run.smithi120:> uname -m 2024-03-28T15:30:15.408 INFO:teuthology.orchestra.run.smithi120.stdout:x86_64 2024-03-28T15:30:15.408 DEBUG:teuthology.orchestra.run.smithi120:> cat /etc/os-release 2024-03-28T15:30:15.464 INFO:teuthology.orchestra.run.smithi120.stdout:NAME="CentOS Stream" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION="9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:ID="centos" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:ID_LIKE="rhel fedora" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:VERSION_ID="9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:PLATFORM_ID="platform:el9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:ANSI_COLOR="0;31" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:LOGO="fedora-logo-icon" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:HOME_URL="https://centos.org/" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T15:30:15.465 INFO:teuthology.orchestra.run.smithi120.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T15:30:15.466 INFO:teuthology.lock.ops:Updating smithi120.front.sepia.ceph.com on lock server 2024-03-28T15:30:15.490 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T15:30:15.500 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T15:30:15.536 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T15:30:15.536 DEBUG:teuthology.orchestra.run.smithi023:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:30:15.539 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:30:15.541 DEBUG:teuthology.orchestra.run.smithi120:> test '!' -e /home/ubuntu/cephtest 2024-03-28T15:30:15.557 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T15:30:15.563 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T15:30:15.563 DEBUG:teuthology.orchestra.run.smithi023:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:30:15.596 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:30:15.598 DEBUG:teuthology.orchestra.run.smithi120:> test -z $(ls -A /var/lib/ceph) 2024-03-28T15:30:15.627 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T15:30:15.700 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T15:30:15.714 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-28T15:30:15.714 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-28T15:30:15.714 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-28T15:30:15.715 DEBUG:teuthology.orchestra.run.smithi023:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:30:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:30:15.732 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-28T15:30:15.790 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-361.el9.x86_64 2024-03-28T15:30:15.791 INFO:teuthology.task.kernel:Running kernel on smithi023: 5.14.0-361.el9.x86_64 2024-03-28T15:30:15.791 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum install -y kernel 2024-03-28T15:30:17.873 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T15:30:19.323 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-28T15:30:28.713 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 910 kB/s | 8.0 MB 00:09 2024-03-28T15:30:30.750 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-28T15:30:35.381 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-28T15:30:36.765 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 75 kB/s | 16 kB 00:00 2024-03-28T15:30:38.566 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-28T15:30:43.742 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-28T15:30:45.915 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T15:30:45.915 INFO:teuthology.orchestra.run.smithi023.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T15:30:46.008 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-28T15:30:46.014 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T15:30:46.014 INFO:teuthology.orchestra.run.smithi023.stdout: Package Architecture Version Repository Size 2024-03-28T15:30:46.014 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:================================================================================ 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:Install 4 Packages 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:30:46.015 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 76 M 2024-03-28T15:30:46.016 INFO:teuthology.orchestra.run.smithi023.stdout:Installed size: 123 M 2024-03-28T15:30:46.016 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-28T15:30:46.576 INFO:teuthology.orchestra.run.smithi023.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 276 kB/s | 66 kB 00:00 2024-03-28T15:30:48.084 INFO:teuthology.orchestra.run.smithi023.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 8.5 MB/s | 15 MB 00:01 2024-03-28T15:30:49.143 INFO:teuthology.orchestra.run.smithi023.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-28T15:30:49.710 INFO:teuthology.orchestra.run.smithi023.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 9.9 MB/s | 33 MB 00:03 2024-03-28T15:30:49.712 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:30:49.713 INFO:teuthology.orchestra.run.smithi023.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-28T15:30:50.406 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-28T15:30:50.839 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-28T15:30:50.839 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-28T15:30:51.861 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-28T15:30:51.861 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-28T15:30:54.466 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-28T15:30:55.127 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:30:55.218 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:30:58.085 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:30:58.221 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:31:03.296 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:31:03.338 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:08.442 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:39.063 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:39.063 INFO:teuthology.orchestra.run.smithi023.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T15:31:39.063 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:31:39.109 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:39.439 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T15:31:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T15:31:39.440 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T15:31:39.874 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T15:31:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T15:31:39.875 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:31:39.875 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-28T15:31:40.180 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:31:40.280 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:31:40.280 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:31:40.280 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T15:31:40.280 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T15:31:40.280 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T15:31:40.280 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-28T15:31:40.281 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi023.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T15:31:40.281 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T15:31:40.363 INFO:teuthology.orchestra.run.smithi023.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T15:31:40.363 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T15:31:40.364 DEBUG:teuthology.orchestra.run.smithi023:> sudo rpm -qi grub2-tools 2024-03-28T15:31:40.401 INFO:teuthology.orchestra.run.smithi023.stdout:Name : grub2-tools 2024-03-28T15:31:40.401 INFO:teuthology.orchestra.run.smithi023.stdout:Epoch : 1 2024-03-28T15:31:40.401 INFO:teuthology.orchestra.run.smithi023.stdout:Version : 2.06 2024-03-28T15:31:40.401 INFO:teuthology.orchestra.run.smithi023.stdout:Release : 68.el9 2024-03-28T15:31:40.401 INFO:teuthology.orchestra.run.smithi023.stdout:Architecture: x86_64 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Group : Unspecified 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Size : 8264254 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:License : GPLv3+ 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Packager : builder@centos.org 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Vendor : CentOS 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T15:31:40.402 INFO:teuthology.orchestra.run.smithi023.stdout:Summary : Support tools for GRUB. 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:Description : 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:hardware devices. 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:31:40.403 INFO:teuthology.orchestra.run.smithi023.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T15:31:40.404 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T15:31:40.404 INFO:teuthology.task.kernel:Updating grub on smithi023 to boot 5.14.0-432.el9.x86_64 2024-03-28T15:31:40.404 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T15:31:41.026 INFO:teuthology.orchestra.run.smithi023.stderr:Generating grub configuration file ... 2024-03-28T15:31:42.530 INFO:teuthology.orchestra.run.smithi023.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T15:31:42.558 INFO:teuthology.orchestra.run.smithi023.stderr:done 2024-03-28T15:31:42.560 DEBUG:teuthology.orchestra.run.smithi023:> mktemp 2024-03-28T15:31:42.576 INFO:teuthology.orchestra.run.smithi023.stdout:/tmp/tmp.OB3Qv2BUDO 2024-03-28T15:31:42.576 DEBUG:teuthology.orchestra.run.smithi023:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OB3Qv2BUDO 2024-03-28T15:31:42.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0666 /tmp/tmp.OB3Qv2BUDO 2024-03-28T15:31:42.799 DEBUG:teuthology.orchestra.remote:smithi023:/tmp/tmp.OB3Qv2BUDO is 6KB 2024-03-28T15:31:42.848 DEBUG:teuthology.orchestra.run.smithi023:> rm -fr /tmp/tmp.OB3Qv2BUDO 2024-03-28T15:31:42.862 DEBUG:teuthology.orchestra.run.smithi023:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T15:31:42.931 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T15:31:42.931 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T15:31:42.931 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T15:31:42.932 INFO:teuthology.orchestra.run.smithi023.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T15:31:42.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T15:31:43.077 DEBUG:teuthology.orchestra.run.smithi023:> sudo shutdown -r now 2024-03-28T15:32:13.080 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-28T15:32:13.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:32:13.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:32:47.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-28T15:32:56.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:32:56.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:32:59.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.23 2024-03-28T15:33:11.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:33:11.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:34:11.842 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T15:34:26.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:34:26.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:34:27.157 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-28T15:34:27.534 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:34:27.534 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-28T15:34:27.535 DEBUG:teuthology.orchestra.run.smithi023:> uname -r 2024-03-28T15:34:27.549 INFO:teuthology.orchestra.run.smithi023.stdout:5.14.0-432.el9.x86_64 2024-03-28T15:34:27.549 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T15:34:27.550 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T15:34:27.550 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T15:34:28.551 DEBUG:teuthology.parallel:result is None 2024-03-28T15:34:28.551 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T15:34:28.560 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T15:34:28.560 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:34:28.563 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:34:28.565 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T15:34:28.583 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T15:34:28.592 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T15:34:28.606 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T15:34:28.606 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:34:28.620 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:34:28.623 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T15:34:28.673 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T15:34:28.684 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T15:34:28.684 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:34:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:34:28.713 DEBUG:teuthology.orchestra.run.smithi120:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T15:34:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:34:28.729 DEBUG:teuthology.orchestra.run.smithi023:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:34:28.732 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:34:28.755 DEBUG:teuthology.orchestra.run.smithi120:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T15:34:28.787 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.808 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.809 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.823 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.828 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.859 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T15:34:28.861 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T15:34:28.870 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T15:34:28.870 DEBUG:teuthology.orchestra.run.smithi023:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:34:28.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:34:28.875 DEBUG:teuthology.orchestra.run.smithi120:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T15:34:28.905 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T15:34:28.919 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T15:34:28.919 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:34:28.944 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:34:28.946 DEBUG:teuthology.orchestra.run.smithi120:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T15:34:28.963 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.043 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.094 DEBUG:teuthology.orchestra.run.smithi023:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.176 DEBUG:teuthology.orchestra.run.smithi023:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.220 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:34:29.220 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:34:29.287 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.387 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.514 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:34:29.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:34:29.583 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T15:34:29.684 DEBUG:teuthology.orchestra.run.smithi120:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.758 DEBUG:teuthology.orchestra.run.smithi120:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T15:34:29.811 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:34:29.811 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T15:34:29.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo service rsyslog restart 2024-03-28T15:34:29.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-28T15:34:29.889 DEBUG:teuthology.orchestra.run.smithi120:> sudo service rsyslog restart 2024-03-28T15:34:29.931 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:34:29.936 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:34:29.962 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T15:34:30.369 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T15:34:30.377 INFO:teuthology.task.internal:Starting timer... 2024-03-28T15:34:30.377 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T15:34:30.393 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T15:34:30.402 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-28T15:34:30.403 DEBUG:teuthology.orchestra.run.smithi023:> sudo service auditd rotate 2024-03-28T15:34:30.460 INFO:teuthology.orchestra.run.smithi023.stdout:Rotating logs: 2024-03-28T15:34:30.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-28T15:34:30.522 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-28T15:34:30.524 DEBUG:teuthology.orchestra.run.smithi120:> sudo service auditd rotate 2024-03-28T15:34:30.578 INFO:teuthology.orchestra.run.smithi120.stdout:Rotating logs: 2024-03-28T15:34:30.580 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T15:34:30.580 DEBUG:teuthology.orchestra.run.smithi023:> /usr/sbin/getenforce 2024-03-28T15:34:30.605 INFO:teuthology.orchestra.run.smithi023.stdout:Permissive 2024-03-28T15:34:30.605 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-28T15:34:30.633 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-28T15:34:30.634 DEBUG:teuthology.orchestra.run.smithi120:> /usr/sbin/getenforce 2024-03-28T15:34:30.658 INFO:teuthology.orchestra.run.smithi120.stdout:Permissive 2024-03-28T15:34:30.659 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi023.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi120.front.sepia.ceph.com': 'permissive'} 2024-03-28T15:34:30.659 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:34:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:34:30.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:34:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:34:30.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T15:34:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T15:34:30.749 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T15:34:30.749 DEBUG:teuthology.orchestra.run.smithi023:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:34:30.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:34:30.843 DEBUG:teuthology.orchestra.run.smithi120:> sudo /usr/sbin/setenforce permissive 2024-03-28T15:34:30.884 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T15:34:30.897 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T15:34:30.903 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T15:34:31.027 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T15:34:31.046 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T15:34:31.047 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi023.front.sepia.ceph.com,smithi117.front.sepia.ceph.com,smithi120.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T15:39:30.621 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi023.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi120.front.sepia.ceph.com')] 2024-03-28T15:39:30.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:39:30.623 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi023.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:30.712 DEBUG:teuthology.orchestra.run.smithi023:> true 2024-03-28T15:39:30.788 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi023.front.sepia.ceph.com' 2024-03-28T15:39:30.789 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-28T15:39:30.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:30.879 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-28T15:39:30.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-28T15:39:30.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-28T15:39:30.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi120.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T15:39:31.043 DEBUG:teuthology.orchestra.run.smithi120:> true 2024-03-28T15:39:31.109 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi120.front.sepia.ceph.com' 2024-03-28T15:39:31.109 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T15:39:31.119 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T15:39:31.119 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:39:31.120 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:39:31.122 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:39:31.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:39:31.125 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T15:39:31.125 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T15:39:31.151 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:39:31.153 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:39:31.167 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:39:31.170 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:39:31.180 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T15:39:31.196 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T15:39:31.196 INFO:teuthology.orchestra.run.smithi023.stderr:sudo: ntpd: command not found 2024-03-28T15:39:31.203 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-28T15:39:31.210 INFO:teuthology.orchestra.run.smithi023.stdout:506 Cannot talk to daemon 2024-03-28T15:39:31.214 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-28T15:39:31.225 INFO:teuthology.orchestra.run.smithi120.stderr:sudo: ntpd: command not found 2024-03-28T15:39:31.226 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:39:31.231 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:39:31.239 INFO:teuthology.orchestra.run.smithi120.stdout:506 Cannot talk to daemon 2024-03-28T15:39:31.242 INFO:teuthology.orchestra.run.smithi023.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:39:31.247 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:39:31.255 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T15:39:31.271 INFO:teuthology.orchestra.run.smithi120.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T15:39:31.291 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:39:31.295 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:39:31.295 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-28T15:39:31.295 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.295 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.296 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.296 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.298 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:39:31.302 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:39:31.302 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-28T15:39:31.302 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.302 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.303 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.303 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.314 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-28T15:39:31.318 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T15:39:31.319 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-28T15:39:31.319 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.319 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.319 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.319 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T15:39:31.320 INFO:teuthology.run_tasks:Running task install... 2024-03-28T15:39:31.332 DEBUG:teuthology.task.install:project ceph 2024-03-28T15:39:31.333 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-28T15:39:31.333 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-28T15:39:31.333 INFO:teuthology.task.install:Using flavor: default 2024-03-28T15:39:31.347 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T15:39:31.347 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-28T15:39:31.348 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:39:31.349 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:39:31.351 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:39:31.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:39:31.352 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-28T15:39:31.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:39:31.501 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:39:31.501 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:39:31.586 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:39:31.586 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:39:31.590 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-28T15:39:31.590 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-28T15:39:31.612 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:39:31.613 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:39:31.613 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:39:31.640 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:39:31.641 DEBUG:teuthology.orchestra.run.smithi023:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:39:31.708 DEBUG:teuthology.orchestra.run.smithi023:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:39:31.755 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:39:31.755 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:39:31.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:39:31.766 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T15:39:31.766 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:39:31.766 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T15:39:31.783 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:39:31.783 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:39:31.788 DEBUG:teuthology.orchestra.run.smithi023:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:39:31.794 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-28T15:39:31.794 DEBUG:teuthology.orchestra.run.smithi120:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T15:39:31.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:39:31.854 INFO:teuthology.orchestra.run.smithi023.stdout:check_obsoletes = 1 2024-03-28T15:39:31.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum clean all 2024-03-28T15:39:31.862 DEBUG:teuthology.orchestra.run.smithi120:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T15:39:31.930 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:39:31.942 DEBUG:teuthology.orchestra.run.smithi120:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T15:39:31.997 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-28T15:39:31.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-28T15:39:32.008 INFO:teuthology.orchestra.run.smithi120.stdout:check_obsoletes = 1 2024-03-28T15:39:32.009 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum clean all 2024-03-28T15:39:32.111 INFO:teuthology.orchestra.run.smithi023.stdout:74 files removed 2024-03-28T15:39:32.137 DEBUG:teuthology.orchestra.run.smithi023:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:39:32.234 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-28T15:39:32.242 INFO:teuthology.orchestra.run.smithi120.stdout:74 files removed 2024-03-28T15:39:32.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:39:32.269 DEBUG:teuthology.orchestra.run.smithi120:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-28T15:39:32.883 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 247 kB/s | 83 kB 00:00 2024-03-28T15:39:32.894 INFO:teuthology.orchestra.run.smithi023.stdout:ceph packages for x86_64 183 kB/s | 83 kB 00:00 2024-03-28T15:39:32.967 INFO:teuthology.orchestra.run.smithi120.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-03-28T15:39:33.250 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-28T15:39:33.261 INFO:teuthology.orchestra.run.smithi023.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-28T15:39:33.384 INFO:teuthology.orchestra.run.smithi120.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-28T15:39:33.626 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-28T15:39:33.629 INFO:teuthology.orchestra.run.smithi023.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-28T15:39:33.785 INFO:teuthology.orchestra.run.smithi120.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-28T15:39:34.031 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-28T15:39:34.078 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-28T15:39:34.187 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-28T15:39:35.445 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2024-03-28T15:39:35.813 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-28T15:39:35.849 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-28T15:39:36.303 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-28T15:39:36.770 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-28T15:39:37.989 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-28T15:39:38.829 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-28T15:39:42.523 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-28T15:39:42.722 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-28T15:39:44.289 INFO:teuthology.orchestra.run.smithi120.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-28T15:39:45.307 INFO:teuthology.orchestra.run.smithi120.stdout:Extra Packages for Enterprise Linux 23 MB/s | 21 MB 00:00 2024-03-28T15:39:46.943 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-03-28T15:39:48.608 INFO:teuthology.orchestra.run.smithi023.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-28T15:39:50.004 INFO:teuthology.orchestra.run.smithi023.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T15:39:50.357 INFO:teuthology.orchestra.run.smithi120.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-28T15:39:52.742 INFO:teuthology.orchestra.run.smithi120.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:39:52.743 INFO:teuthology.orchestra.run.smithi120.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:39:52.755 INFO:teuthology.orchestra.run.smithi120.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:39:52.756 INFO:teuthology.orchestra.run.smithi120.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:39:52.757 INFO:teuthology.orchestra.run.smithi120.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:39:52.758 INFO:teuthology.orchestra.run.smithi120.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:39:52.760 INFO:teuthology.orchestra.run.smithi120.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:39:52.762 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:39:52.763 INFO:teuthology.orchestra.run.smithi120.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:39:52.843 INFO:teuthology.orchestra.run.smithi120.stdout:Dependencies resolved. 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: Package Arch Version Repository Size 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout:Installing: 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:39:52.870 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:39:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:39:52.872 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrading: 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:39:52.873 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:39:52.874 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout:Installing dependencies: 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:39:52.875 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:39:52.876 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:39:52.877 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:39:52.878 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:39:52.879 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:39:52.880 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:39:52.881 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:39:52.882 INFO:teuthology.orchestra.run.smithi120.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Installing weak dependencies: 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction Summary 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:============================================================================================ 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Install 150 Packages 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout:Upgrade 24 Packages 2024-03-28T15:39:52.883 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:39:52.884 INFO:teuthology.orchestra.run.smithi120.stdout:Total download size: 225 M 2024-03-28T15:39:52.884 INFO:teuthology.orchestra.run.smithi120.stdout:Downloading Packages: 2024-03-28T15:39:53.675 INFO:teuthology.orchestra.run.smithi120.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-03-28T15:39:54.050 INFO:teuthology.orchestra.run.smithi120.stdout:(2/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 8.9 MB/s | 5.4 MB 00:00 2024-03-28T15:39:54.117 INFO:teuthology.orchestra.run.smithi120.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 2.1 MB/s | 145 kB 00:00 2024-03-28T15:39:54.375 INFO:teuthology.orchestra.run.smithi120.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 8.9 MB/s | 2.3 MB 00:00 2024-03-28T15:39:54.517 INFO:teuthology.orchestra.run.smithi120.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-28T15:39:54.717 INFO:teuthology.orchestra.run.smithi120.stdout:(6/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 854 kB/s | 890 kB 00:01 2024-03-28T15:39:54.909 INFO:teuthology.orchestra.run.smithi120.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 4.7 MB 00:00 2024-03-28T15:39:55.118 INFO:teuthology.orchestra.run.smithi120.stdout:(8/174): ceph-common-19.0.0-2145.g3be9a5c9.el9. 12 MB/s | 20 MB 00:01 2024-03-28T15:39:55.168 INFO:teuthology.orchestra.run.smithi120.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 505 kB/s | 25 kB 00:00 2024-03-28T15:39:55.185 INFO:teuthology.orchestra.run.smithi023.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-03-28T15:39:55.585 INFO:teuthology.orchestra.run.smithi120.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 15 MB/s | 9.9 MB 00:00 2024-03-28T15:39:55.643 INFO:teuthology.orchestra.run.smithi120.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.7 MB/s | 220 kB 00:00 2024-03-28T15:39:55.702 INFO:teuthology.orchestra.run.smithi120.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 461 kB/s | 27 kB 00:00 2024-03-28T15:39:55.761 INFO:teuthology.orchestra.run.smithi120.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 536 kB/s | 31 kB 00:00 2024-03-28T15:39:55.861 INFO:teuthology.orchestra.run.smithi120.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 7.3 MB/s | 746 kB 00:00 2024-03-28T15:39:55.928 INFO:teuthology.orchestra.run.smithi120.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.4 MB/s | 163 kB 00:00 2024-03-28T15:39:55.995 INFO:teuthology.orchestra.run.smithi120.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.9 MB/s | 127 kB 00:00 2024-03-28T15:39:56.079 INFO:teuthology.orchestra.run.smithi120.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 5.9 MB/s | 501 kB 00:00 2024-03-28T15:39:56.512 INFO:teuthology.orchestra.run.smithi120.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 12 MB/s | 5.1 MB 00:00 2024-03-28T15:39:56.579 INFO:teuthology.orchestra.run.smithi120.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 678 kB/s | 45 kB 00:00 2024-03-28T15:39:56.654 INFO:teuthology.orchestra.run.smithi120.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.7 MB/s | 131 kB 00:00 2024-03-28T15:39:56.721 INFO:teuthology.orchestra.run.smithi120.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-28T15:39:56.797 INFO:teuthology.orchestra.run.smithi120.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 4.2 MB/s | 321 kB 00:00 2024-03-28T15:39:56.872 INFO:teuthology.orchestra.run.smithi120.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T15:39:56.939 INFO:teuthology.orchestra.run.smithi120.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-28T15:39:57.006 INFO:teuthology.orchestra.run.smithi120.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-28T15:39:57.281 INFO:teuthology.orchestra.run.smithi120.stdout:(26/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 11 MB/s | 3.1 MB 00:00 2024-03-28T15:39:57.340 INFO:teuthology.orchestra.run.smithi120.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-28T15:39:57.407 INFO:teuthology.orchestra.run.smithi120.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 508 kB/s | 34 kB 00:00 2024-03-28T15:39:57.474 INFO:teuthology.orchestra.run.smithi120.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-28T15:39:57.704 INFO:teuthology.orchestra.run.smithi023.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:39:57.705 INFO:teuthology.orchestra.run.smithi023.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:39:57.721 INFO:teuthology.orchestra.run.smithi023.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:39:57.722 INFO:teuthology.orchestra.run.smithi023.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:39:57.723 INFO:teuthology.orchestra.run.smithi023.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:39:57.725 INFO:teuthology.orchestra.run.smithi023.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:39:57.727 INFO:teuthology.orchestra.run.smithi023.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:39:57.730 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:39:57.731 INFO:teuthology.orchestra.run.smithi023.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:39:57.817 INFO:teuthology.orchestra.run.smithi023.stdout:Dependencies resolved. 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout: Package Arch Version Repository Size 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout:Installing: 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:39:57.830 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:39:57.831 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:39:57.832 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrading: 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:39:57.833 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:39:57.834 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout:Installing dependencies: 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:39:57.835 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:39:57.836 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:39:57.837 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:39:57.838 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:39:57.839 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:39:57.840 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:39:57.841 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:39:57.842 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout:Installing weak dependencies: 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction Summary 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout:============================================================================================ 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout:Install 150 Packages 2024-03-28T15:39:57.843 INFO:teuthology.orchestra.run.smithi023.stdout:Upgrade 24 Packages 2024-03-28T15:39:57.844 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:39:57.844 INFO:teuthology.orchestra.run.smithi023.stdout:Total download size: 225 M 2024-03-28T15:39:57.844 INFO:teuthology.orchestra.run.smithi023.stdout:Downloading Packages: 2024-03-28T15:39:57.932 INFO:teuthology.orchestra.run.smithi120.stdout:(30/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 17 MB/s | 48 MB 00:02 2024-03-28T15:39:57.982 INFO:teuthology.orchestra.run.smithi120.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 3.5 MB/s | 1.8 MB 00:00 2024-03-28T15:39:58.058 INFO:teuthology.orchestra.run.smithi120.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.2 MB/s | 248 kB 00:00 2024-03-28T15:39:58.116 INFO:teuthology.orchestra.run.smithi120.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 848 kB/s | 49 kB 00:00 2024-03-28T15:39:58.175 INFO:teuthology.orchestra.run.smithi120.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 282 kB/s | 16 kB 00:00 2024-03-28T15:39:58.250 INFO:teuthology.orchestra.run.smithi120.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.4 MB/s | 264 kB 00:00 2024-03-28T15:39:58.258 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 925 kB/s | 19 MB 00:20 2024-03-28T15:39:58.376 INFO:teuthology.orchestra.run.smithi120.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 17 MB/s | 7.4 MB 00:00 2024-03-28T15:39:58.409 INFO:teuthology.orchestra.run.smithi120.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-28T15:39:58.426 INFO:teuthology.orchestra.run.smithi120.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 365 kB/s | 18 kB 00:00 2024-03-28T15:39:58.568 INFO:teuthology.orchestra.run.smithi120.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-03-28T15:39:58.610 INFO:teuthology.orchestra.run.smithi120.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 920 kB 00:00 2024-03-28T15:39:58.635 INFO:teuthology.orchestra.run.smithi120.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-28T15:39:58.660 INFO:teuthology.orchestra.run.smithi120.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-28T15:39:58.686 INFO:teuthology.orchestra.run.smithi120.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-28T15:39:58.744 INFO:teuthology.orchestra.run.smithi023.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2024-03-28T15:39:58.786 INFO:teuthology.orchestra.run.smithi120.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-03-28T15:39:58.812 INFO:teuthology.orchestra.run.smithi120.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 137 kB/s | 17 kB 00:00 2024-03-28T15:39:58.828 INFO:teuthology.orchestra.run.smithi120.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-28T15:39:58.854 INFO:teuthology.orchestra.run.smithi120.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-28T15:39:58.887 INFO:teuthology.orchestra.run.smithi120.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 760 kB/s | 44 kB 00:00 2024-03-28T15:39:58.920 INFO:teuthology.orchestra.run.smithi120.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-28T15:39:58.944 INFO:teuthology.orchestra.run.smithi023.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.4 MB/s | 890 kB 00:00 2024-03-28T15:39:58.946 INFO:teuthology.orchestra.run.smithi120.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-28T15:39:58.972 INFO:teuthology.orchestra.run.smithi120.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 936 kB/s | 46 kB 00:00 2024-03-28T15:39:58.986 INFO:teuthology.orchestra.run.smithi023.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 3.4 MB/s | 145 kB 00:00 2024-03-28T15:39:58.997 INFO:teuthology.orchestra.run.smithi120.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T15:39:59.030 INFO:teuthology.orchestra.run.smithi120.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-28T15:39:59.056 INFO:teuthology.orchestra.run.smithi120.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-03-28T15:39:59.081 INFO:teuthology.orchestra.run.smithi120.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-28T15:39:59.170 INFO:teuthology.orchestra.run.smithi023.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 2.3 MB 00:00 2024-03-28T15:39:59.240 INFO:teuthology.orchestra.run.smithi120.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-28T15:39:59.273 INFO:teuthology.orchestra.run.smithi120.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-28T15:39:59.298 INFO:teuthology.orchestra.run.smithi120.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-28T15:39:59.303 INFO:teuthology.orchestra.run.smithi023.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-28T15:39:59.524 INFO:teuthology.orchestra.run.smithi120.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-28T15:39:59.541 INFO:teuthology.orchestra.run.smithi120.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 101 kB/s | 24 kB 00:00 2024-03-28T15:39:59.558 INFO:teuthology.orchestra.run.smithi120.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-28T15:39:59.583 INFO:teuthology.orchestra.run.smithi120.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-28T15:39:59.601 INFO:teuthology.orchestra.run.smithi120.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-03-28T15:39:59.654 INFO:teuthology.orchestra.run.smithi023.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 4.7 MB 00:00 2024-03-28T15:39:59.792 INFO:teuthology.orchestra.run.smithi120.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2024-03-28T15:39:59.817 INFO:teuthology.orchestra.run.smithi120.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2024-03-28T15:39:59.842 INFO:teuthology.orchestra.run.smithi120.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-28T15:39:59.845 INFO:teuthology.orchestra.run.smithi023.stdout:(7/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 4.2 MB/s | 5.4 MB 00:01 2024-03-28T15:39:59.868 INFO:teuthology.orchestra.run.smithi120.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-28T15:39:59.884 INFO:teuthology.orchestra.run.smithi120.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 994 kB/s | 41 kB 00:00 2024-03-28T15:39:59.910 INFO:teuthology.orchestra.run.smithi120.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-28T15:39:59.944 INFO:teuthology.orchestra.run.smithi120.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T15:39:59.969 INFO:teuthology.orchestra.run.smithi120.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-28T15:39:59.986 INFO:teuthology.orchestra.run.smithi120.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T15:40:00.003 INFO:teuthology.orchestra.run.smithi120.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T15:40:00.386 INFO:teuthology.orchestra.run.smithi120.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 101 kB/s | 39 kB 00:00 2024-03-28T15:40:00.813 INFO:teuthology.orchestra.run.smithi023.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 15 MB/s | 17 MB 00:01 2024-03-28T15:40:00.836 INFO:teuthology.orchestra.run.smithi120.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-03-28T15:40:00.853 INFO:teuthology.orchestra.run.smithi120.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 88 kB/s | 41 kB 00:00 2024-03-28T15:40:00.887 INFO:teuthology.orchestra.run.smithi120.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T15:40:00.904 INFO:teuthology.orchestra.run.smithi120.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T15:40:00.920 INFO:teuthology.orchestra.run.smithi120.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-28T15:40:00.946 INFO:teuthology.orchestra.run.smithi120.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-28T15:40:00.971 INFO:teuthology.orchestra.run.smithi120.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T15:40:00.996 INFO:teuthology.orchestra.run.smithi120.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T15:40:01.055 INFO:teuthology.orchestra.run.smithi120.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T15:40:01.089 INFO:teuthology.orchestra.run.smithi120.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T15:40:01.114 INFO:teuthology.orchestra.run.smithi120.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-28T15:40:01.131 INFO:teuthology.orchestra.run.smithi120.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 4.4 MB/s | 187 kB 00:00 2024-03-28T15:40:01.363 INFO:teuthology.orchestra.run.smithi023.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 46 kB/s | 25 kB 00:00 2024-03-28T15:40:01.382 INFO:teuthology.orchestra.run.smithi120.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 702 kB/s | 187 kB 00:00 2024-03-28T15:40:01.407 INFO:teuthology.orchestra.run.smithi120.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 803 kB 00:00 2024-03-28T15:40:01.457 INFO:teuthology.orchestra.run.smithi120.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-03-28T15:40:01.482 INFO:teuthology.orchestra.run.smithi120.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-28T15:40:01.541 INFO:teuthology.orchestra.run.smithi120.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 257 kB/s | 21 kB 00:00 2024-03-28T15:40:01.566 INFO:teuthology.orchestra.run.smithi120.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-03-28T15:40:01.974 INFO:teuthology.orchestra.run.smithi120.stdout:(93/174): postgresql-13.14-1.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-03-28T15:40:02.266 INFO:teuthology.orchestra.run.smithi120.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 3.1 MB/s | 911 kB 00:00 2024-03-28T15:40:02.342 INFO:teuthology.orchestra.run.smithi120.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-03-28T15:40:02.750 INFO:teuthology.orchestra.run.smithi120.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 3.4 MB/s | 4.1 MB 00:01 2024-03-28T15:40:02.859 INFO:teuthology.orchestra.run.smithi120.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-28T15:40:02.976 INFO:teuthology.orchestra.run.smithi120.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.1 MB/s | 248 kB 00:00 2024-03-28T15:40:03.092 INFO:teuthology.orchestra.run.smithi120.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.9 MB/s | 221 kB 00:00 2024-03-28T15:40:03.168 INFO:teuthology.orchestra.run.smithi120.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 478 kB/s | 36 kB 00:00 2024-03-28T15:40:03.234 INFO:teuthology.orchestra.run.smithi120.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 712 kB/s | 47 kB 00:00 2024-03-28T15:40:03.301 INFO:teuthology.orchestra.run.smithi120.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-28T15:40:03.360 INFO:teuthology.orchestra.run.smithi120.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 382 kB/s | 22 kB 00:00 2024-03-28T15:40:03.602 INFO:teuthology.orchestra.run.smithi120.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-03-28T15:40:03.786 INFO:teuthology.orchestra.run.smithi120.stdout:(105/174): postgresql-server-13.14-1.el9.x86_64 4.0 MB/s | 5.8 MB 00:01 2024-03-28T15:40:03.933 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 6.1 MB 00:02 2024-03-28T15:40:03.969 INFO:teuthology.orchestra.run.smithi120.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-03-28T15:40:04.011 INFO:teuthology.orchestra.run.smithi120.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-03-28T15:40:04.028 INFO:teuthology.orchestra.run.smithi120.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-28T15:40:04.686 INFO:teuthology.orchestra.run.smithi120.stdout:(109/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-03-28T15:40:04.703 INFO:teuthology.orchestra.run.smithi120.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 38 kB/s | 26 kB 00:00 2024-03-28T15:40:04.770 INFO:teuthology.orchestra.run.smithi120.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-03-28T15:40:04.796 INFO:teuthology.orchestra.run.smithi120.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 734 kB/s | 67 kB 00:00 2024-03-28T15:40:04.854 INFO:teuthology.orchestra.run.smithi120.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-28T15:40:04.996 INFO:teuthology.orchestra.run.smithi120.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-28T15:40:05.055 INFO:teuthology.orchestra.run.smithi120.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-03-28T15:40:05.080 INFO:teuthology.orchestra.run.smithi120.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 524 kB/s | 43 kB 00:00 2024-03-28T15:40:05.114 INFO:teuthology.orchestra.run.smithi120.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 552 kB/s | 32 kB 00:00 2024-03-28T15:40:05.121 INFO:teuthology.orchestra.run.smithi023.stdout:(10/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 1.9 MB/s | 9.9 MB 00:05 2024-03-28T15:40:05.139 INFO:teuthology.orchestra.run.smithi120.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 256 kB/s | 15 kB 00:00 2024-03-28T15:40:05.181 INFO:teuthology.orchestra.run.smithi120.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-28T15:40:05.214 INFO:teuthology.orchestra.run.smithi120.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-28T15:40:05.258 INFO:teuthology.orchestra.run.smithi120.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.8 MB/s | 303 kB 00:00 2024-03-28T15:40:05.292 INFO:teuthology.orchestra.run.smithi120.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 455 kB/s | 35 kB 00:00 2024-03-28T15:40:05.342 INFO:teuthology.orchestra.run.smithi120.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-28T15:40:05.371 INFO:teuthology.orchestra.run.smithi023.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 883 kB/s | 220 kB 00:00 2024-03-28T15:40:05.505 INFO:teuthology.orchestra.run.smithi023.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 203 kB/s | 27 kB 00:00 2024-03-28T15:40:05.584 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-28T15:40:05.622 INFO:teuthology.orchestra.run.smithi023.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 268 kB/s | 31 kB 00:00 2024-03-28T15:40:05.709 INFO:teuthology.orchestra.run.smithi120.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.9 MB/s | 2.0 MB 00:00 2024-03-28T15:40:05.734 INFO:teuthology.orchestra.run.smithi120.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 368 kB/s | 144 kB 00:00 2024-03-28T15:40:05.768 INFO:teuthology.orchestra.run.smithi120.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-03-28T15:40:05.794 INFO:teuthology.orchestra.run.smithi120.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 300 kB/s | 18 kB 00:00 2024-03-28T15:40:05.835 INFO:teuthology.orchestra.run.smithi120.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 347 kB/s | 23 kB 00:00 2024-03-28T15:40:05.869 INFO:teuthology.orchestra.run.smithi120.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-03-28T15:40:05.894 INFO:teuthology.orchestra.run.smithi120.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-03-28T15:40:06.061 INFO:teuthology.orchestra.run.smithi120.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-28T15:40:06.087 INFO:teuthology.orchestra.run.smithi120.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 241 kB/s | 46 kB 00:00 2024-03-28T15:40:06.097 INFO:teuthology.orchestra.run.smithi023.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 1.5 MB/s | 746 kB 00:00 2024-03-28T15:40:06.120 INFO:teuthology.orchestra.run.smithi120.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-28T15:40:06.146 INFO:teuthology.orchestra.run.smithi120.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 984 kB/s | 58 kB 00:00 2024-03-28T15:40:06.179 INFO:teuthology.orchestra.run.smithi120.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-28T15:40:06.196 INFO:teuthology.orchestra.run.smithi120.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-03-28T15:40:06.238 INFO:teuthology.orchestra.run.smithi120.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-03-28T15:40:06.255 INFO:teuthology.orchestra.run.smithi120.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-03-28T15:40:06.289 INFO:teuthology.orchestra.run.smithi120.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-03-28T15:40:06.297 INFO:teuthology.orchestra.run.smithi023.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 816 kB/s | 163 kB 00:00 2024-03-28T15:40:06.322 INFO:teuthology.orchestra.run.smithi120.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.8 MB/s | 188 kB 00:00 2024-03-28T15:40:06.348 INFO:teuthology.orchestra.run.smithi120.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-28T15:40:06.381 INFO:teuthology.orchestra.run.smithi023.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.5 MB/s | 127 kB 00:00 2024-03-28T15:40:06.381 INFO:teuthology.orchestra.run.smithi120.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 610 kB/s | 36 kB 00:00 2024-03-28T15:40:06.407 INFO:teuthology.orchestra.run.smithi120.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 870 kB/s | 51 kB 00:00 2024-03-28T15:40:06.440 INFO:teuthology.orchestra.run.smithi120.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-28T15:40:06.474 INFO:teuthology.orchestra.run.smithi120.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-03-28T15:40:06.533 INFO:teuthology.orchestra.run.smithi120.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-03-28T15:40:06.558 INFO:teuthology.orchestra.run.smithi120.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-03-28T15:40:06.592 INFO:teuthology.orchestra.run.smithi120.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-28T15:40:06.731 INFO:teuthology.orchestra.run.smithi023.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 1.4 MB/s | 501 kB 00:00 2024-03-28T15:40:06.800 INFO:teuthology.orchestra.run.smithi120.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-28T15:40:07.015 INFO:teuthology.orchestra.run.smithi023.stdout:(18/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.4 MB/s | 20 MB 00:08 2024-03-28T15:40:07.140 INFO:teuthology.orchestra.run.smithi023.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 362 kB/s | 45 kB 00:00 2024-03-28T15:40:07.348 INFO:teuthology.orchestra.run.smithi023.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 630 kB/s | 131 kB 00:00 2024-03-28T15:40:07.417 INFO:teuthology.orchestra.run.smithi120.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.1 MB/s | 3.4 MB 00:00 2024-03-28T15:40:07.509 INFO:teuthology.orchestra.run.smithi120.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 3.5 MB/s | 321 kB 00:00 2024-03-28T15:40:07.565 INFO:teuthology.orchestra.run.smithi023.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 748 kB/s | 162 kB 00:00 2024-03-28T15:40:07.866 INFO:teuthology.orchestra.run.smithi023.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 321 kB 00:00 2024-03-28T15:40:08.199 INFO:teuthology.orchestra.run.smithi023.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 896 kB/s | 298 kB 00:00 2024-03-28T15:40:08.400 INFO:teuthology.orchestra.run.smithi023.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 498 kB/s | 99 kB 00:00 2024-03-28T15:40:08.500 INFO:teuthology.orchestra.run.smithi023.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 854 kB/s | 85 kB 00:00 2024-03-28T15:40:08.693 INFO:teuthology.orchestra.run.smithi120.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-03-28T15:40:09.012 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 6.2 MB/s | 21 MB 00:03 2024-03-28T15:40:09.135 INFO:teuthology.orchestra.run.smithi120.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 4.2 MB/s | 1.8 MB 00:00 2024-03-28T15:40:09.202 INFO:teuthology.orchestra.run.smithi120.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 635 kB/s | 42 kB 00:00 2024-03-28T15:40:09.286 INFO:teuthology.orchestra.run.smithi120.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.9 MB/s | 248 kB 00:00 2024-03-28T15:40:09.344 INFO:teuthology.orchestra.run.smithi120.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-28T15:40:09.403 INFO:teuthology.orchestra.run.smithi120.stdout:(157/174): NetworkManager-1.47.2-1.el9.x86_64.r 1.2 MB/s | 2.3 MB 00:01 2024-03-28T15:40:09.453 INFO:teuthology.orchestra.run.smithi120.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 335 kB/s | 36 kB 00:00 2024-03-28T15:40:09.478 INFO:teuthology.orchestra.run.smithi120.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 192 kB 00:00 2024-03-28T15:40:09.604 INFO:teuthology.orchestra.run.smithi120.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 184 kB/s | 23 kB 00:00 2024-03-28T15:40:09.687 INFO:teuthology.orchestra.run.smithi120.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-03-28T15:40:09.713 INFO:teuthology.orchestra.run.smithi120.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 986 kB/s | 256 kB 00:00 2024-03-28T15:40:09.796 INFO:teuthology.orchestra.run.smithi120.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-28T15:40:09.846 INFO:teuthology.orchestra.run.smithi120.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 975 kB/s | 155 kB 00:00 2024-03-28T15:40:09.872 INFO:teuthology.orchestra.run.smithi120.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 849 kB/s | 64 kB 00:00 2024-03-28T15:40:09.931 INFO:teuthology.orchestra.run.smithi120.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 343 kB/s | 29 kB 00:00 2024-03-28T15:40:10.106 INFO:teuthology.orchestra.run.smithi120.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 1.2 MB 00:00 2024-03-28T15:40:10.339 INFO:teuthology.orchestra.run.smithi120.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.8 MB/s | 2.3 MB 00:00 2024-03-28T15:40:10.440 INFO:teuthology.orchestra.run.smithi120.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.5 MB/s | 465 kB 00:00 2024-03-28T15:40:10.492 INFO:teuthology.orchestra.run.smithi023.stdout:(26/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.2 MB/s | 48 MB 00:09 2024-03-28T15:40:10.510 INFO:teuthology.orchestra.run.smithi120.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 467 kB/s | 31 kB 00:00 2024-03-28T15:40:10.550 INFO:teuthology.orchestra.run.smithi023.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-28T15:40:10.609 INFO:teuthology.orchestra.run.smithi023.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 585 kB/s | 34 kB 00:00 2024-03-28T15:40:10.657 INFO:teuthology.orchestra.run.smithi120.stdout:(171/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.2 MB/s | 1.1 MB 00:00 2024-03-28T15:40:10.676 INFO:teuthology.orchestra.run.smithi023.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.0 MB/s | 139 kB 00:00 2024-03-28T15:40:10.757 INFO:teuthology.orchestra.run.smithi120.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 744 kB/s | 74 kB 00:00 2024-03-28T15:40:10.759 INFO:teuthology.orchestra.run.smithi023.stdout:(30/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 1.3 MB/s | 5.1 MB 00:04 2024-03-28T15:40:10.966 INFO:teuthology.orchestra.run.smithi120.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 2.1 MB/s | 2.2 MB 00:01 2024-03-28T15:40:11.010 INFO:teuthology.orchestra.run.smithi023.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 1.2 MB/s | 3.1 MB 00:02 2024-03-28T15:40:11.044 INFO:teuthology.orchestra.run.smithi023.stdout:(32/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 4.9 MB/s | 1.8 MB 00:00 2024-03-28T15:40:11.094 INFO:teuthology.orchestra.run.smithi023.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 988 kB/s | 49 kB 00:00 2024-03-28T15:40:11.144 INFO:teuthology.orchestra.run.smithi023.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2145.g3 330 kB/s | 16 kB 00:00 2024-03-28T15:40:11.202 INFO:teuthology.orchestra.run.smithi023.stdout:(35/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 4.4 MB/s | 264 kB 00:00 2024-03-28T15:40:11.228 INFO:teuthology.orchestra.run.smithi023.stdout:(36/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.1 MB/s | 248 kB 00:00 2024-03-28T15:40:11.336 INFO:teuthology.orchestra.run.smithi023.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 5.5 MB/s | 748 kB 00:00 2024-03-28T15:40:11.353 INFO:teuthology.orchestra.run.smithi023.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 145 kB/s | 18 kB 00:00 2024-03-28T15:40:11.587 INFO:teuthology.orchestra.run.smithi023.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 3.9 MB/s | 920 kB 00:00 2024-03-28T15:40:11.621 INFO:teuthology.orchestra.run.smithi023.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-03-28T15:40:11.646 INFO:teuthology.orchestra.run.smithi023.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-28T15:40:11.688 INFO:teuthology.orchestra.run.smithi023.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-28T15:40:11.713 INFO:teuthology.orchestra.run.smithi023.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 472 kB/s | 32 kB 00:00 2024-03-28T15:40:11.805 INFO:teuthology.orchestra.run.smithi023.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-28T15:40:11.864 INFO:teuthology.orchestra.run.smithi023.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-28T15:40:11.889 INFO:teuthology.orchestra.run.smithi023.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 123 kB/s | 10 kB 00:00 2024-03-28T15:40:11.922 INFO:teuthology.orchestra.run.smithi023.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-28T15:40:11.948 INFO:teuthology.orchestra.run.smithi023.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-03-28T15:40:11.981 INFO:teuthology.orchestra.run.smithi023.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-28T15:40:12.015 INFO:teuthology.orchestra.run.smithi023.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-28T15:40:12.040 INFO:teuthology.orchestra.run.smithi023.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 806 kB/s | 46 kB 00:00 2024-03-28T15:40:12.066 INFO:teuthology.orchestra.run.smithi023.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-28T15:40:12.091 INFO:teuthology.orchestra.run.smithi023.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-28T15:40:12.116 INFO:teuthology.orchestra.run.smithi023.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-28T15:40:12.133 INFO:teuthology.orchestra.run.smithi023.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2024-03-28T15:40:12.325 INFO:teuthology.orchestra.run.smithi023.stdout:(56/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-28T15:40:12.392 INFO:teuthology.orchestra.run.smithi023.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-03-28T15:40:12.409 INFO:teuthology.orchestra.run.smithi023.stdout:(58/174): python3-attrs-20.3.0-7.el9.noarch.rpm 935 kB/s | 78 kB 00:00 2024-03-28T15:40:12.441 INFO:teuthology.orchestra.run.smithi120.stdout:(174/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x 973 kB/s | 17 MB 00:17 2024-03-28T15:40:12.449 INFO:teuthology.orchestra.run.smithi120.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:40:12.449 INFO:teuthology.orchestra.run.smithi120.stdout:Total 11 MB/s | 225 MB 00:19 2024-03-28T15:40:12.676 INFO:teuthology.orchestra.run.smithi023.stdout:(59/174): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-28T15:40:12.693 INFO:teuthology.orchestra.run.smithi023.stdout:(60/174): python3-jsonpatch-1.21-16.el9.noarch. 86 kB/s | 24 kB 00:00 2024-03-28T15:40:12.718 INFO:teuthology.orchestra.run.smithi023.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 434 kB/s | 18 kB 00:00 2024-03-28T15:40:12.743 INFO:teuthology.orchestra.run.smithi023.stdout:(62/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-28T15:40:12.760 INFO:teuthology.orchestra.run.smithi023.stdout:(63/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-03-28T15:40:13.011 INFO:teuthology.orchestra.run.smithi023.stdout:(64/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 4.8 MB 00:00 2024-03-28T15:40:13.014 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction check 2024-03-28T15:40:13.036 INFO:teuthology.orchestra.run.smithi023.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 790 kB/s | 217 kB 00:00 2024-03-28T15:40:13.070 INFO:teuthology.orchestra.run.smithi023.stdout:(66/174): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 174 kB 00:00 2024-03-28T15:40:13.095 INFO:teuthology.orchestra.run.smithi023.stdout:(67/174): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 69 kB 00:00 2024-03-28T15:40:13.112 INFO:teuthology.orchestra.run.smithi023.stdout:(68/174): python3-prettytable-0.7.2-27.el9.noar 990 kB/s | 41 kB 00:00 2024-03-28T15:40:13.145 INFO:teuthology.orchestra.run.smithi023.stdout:(69/174): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-03-28T15:40:13.171 INFO:teuthology.orchestra.run.smithi023.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-28T15:40:13.196 INFO:teuthology.orchestra.run.smithi023.stdout:(71/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-28T15:40:13.213 INFO:teuthology.orchestra.run.smithi023.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-28T15:40:13.230 INFO:teuthology.orchestra.run.smithi023.stdout:(73/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-28T15:40:13.284 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction check succeeded. 2024-03-28T15:40:13.284 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction test 2024-03-28T15:40:13.489 INFO:teuthology.orchestra.run.smithi023.stdout:(74/174): python3-toml-0.10.2-6.el9.noarch.rpm 149 kB/s | 39 kB 00:00 2024-03-28T15:40:13.797 INFO:teuthology.orchestra.run.smithi023.stdout:(75/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-03-28T15:40:13.822 INFO:teuthology.orchestra.run.smithi023.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 123 kB/s | 41 kB 00:00 2024-03-28T15:40:13.848 INFO:teuthology.orchestra.run.smithi023.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-28T15:40:13.873 INFO:teuthology.orchestra.run.smithi023.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-28T15:40:13.885 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T15:40:13.898 INFO:teuthology.orchestra.run.smithi023.stdout:(79/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-28T15:40:13.924 INFO:teuthology.orchestra.run.smithi023.stdout:(80/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-28T15:40:13.949 INFO:teuthology.orchestra.run.smithi023.stdout:(81/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-28T15:40:13.974 INFO:teuthology.orchestra.run.smithi023.stdout:(82/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-28T15:40:14.024 INFO:teuthology.orchestra.run.smithi023.stdout:(83/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-28T15:40:14.050 INFO:teuthology.orchestra.run.smithi023.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-28T15:40:14.067 INFO:teuthology.orchestra.run.smithi023.stdout:(85/174): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2024-03-28T15:40:14.100 INFO:teuthology.orchestra.run.smithi023.stdout:(86/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-28T15:40:14.626 INFO:teuthology.orchestra.run.smithi023.stdout:(87/174): jq-1.6-16.el9.x86_64.rpm 335 kB/s | 187 kB 00:00 2024-03-28T15:40:14.826 INFO:teuthology.orchestra.run.smithi023.stdout:(88/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-28T15:40:14.851 INFO:teuthology.orchestra.run.smithi023.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 868 kB/s | 195 kB 00:00 2024-03-28T15:40:14.935 INFO:teuthology.orchestra.run.smithi023.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-03-28T15:40:15.068 INFO:teuthology.orchestra.run.smithi023.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 99 kB/s | 21 kB 00:00 2024-03-28T15:40:15.081 INFO:teuthology.orchestra.run.smithi120.stdout:Transaction test succeeded. 2024-03-28T15:40:15.082 INFO:teuthology.orchestra.run.smithi120.stdout:Running transaction 2024-03-28T15:40:15.144 INFO:teuthology.orchestra.run.smithi023.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 140 kB/s | 25 kB 00:00 2024-03-28T15:40:15.277 INFO:teuthology.orchestra.run.smithi023.stdout:(93/174): ceph-mgr-diskprediction-local-19.0.0- 1.6 MB/s | 7.4 MB 00:04 2024-03-28T15:40:15.527 INFO:teuthology.orchestra.run.smithi023.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 4.1 MB/s | 1.6 MB 00:00 2024-03-28T15:40:15.611 INFO:teuthology.orchestra.run.smithi023.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-03-28T15:40:15.653 INFO:teuthology.orchestra.run.smithi023.stdout:(96/174): postgresql-contrib-13.14-1.el9.x86_64 2.4 MB/s | 911 kB 00:00 2024-03-28T15:40:15.753 INFO:teuthology.orchestra.run.smithi023.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-28T15:40:15.845 INFO:teuthology.orchestra.run.smithi023.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-28T15:40:15.946 INFO:teuthology.orchestra.run.smithi023.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 2.2 MB/s | 221 kB 00:00 2024-03-28T15:40:16.046 INFO:teuthology.orchestra.run.smithi023.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 359 kB/s | 36 kB 00:00 2024-03-28T15:40:16.179 INFO:teuthology.orchestra.run.smithi023.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.7 MB/s | 4.1 MB 00:01 2024-03-28T15:40:16.204 INFO:teuthology.orchestra.run.smithi023.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 299 kB/s | 47 kB 00:00 2024-03-28T15:40:16.236 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:40:16.237 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-28T15:40:16.251 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-28T15:40:16.252 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-28T15:40:16.253 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-28T15:40:16.254 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-28T15:40:16.256 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-28T15:40:16.259 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-28T15:40:16.260 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-28T15:40:16.280 INFO:teuthology.orchestra.run.smithi023.stdout:(103/174): postgresql-server-13.14-1.el9.x86_64 8.6 MB/s | 5.8 MB 00:00 2024-03-28T15:40:16.342 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-28T15:40:16.346 INFO:teuthology.orchestra.run.smithi023.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 561 kB/s | 94 kB 00:00 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-28T15:40:16.359 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-28T15:40:16.360 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-28T15:40:16.361 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-28T15:40:16.362 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-28T15:40:16.363 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-28T15:40:16.364 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T15:40:16.365 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T15:40:16.366 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T15:40:16.367 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T15:40:16.368 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T15:40:16.369 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T15:40:16.370 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T15:40:16.371 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T15:40:16.372 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout:Install 150 Packages 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 24 Packages 2024-03-28T15:40:16.373 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:40:16.374 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 225 M 2024-03-28T15:40:16.374 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-28T15:40:16.455 INFO:teuthology.orchestra.run.smithi023.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2024-03-28T15:40:16.530 INFO:teuthology.orchestra.run.smithi023.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-03-28T15:40:16.831 INFO:teuthology.orchestra.run.smithi023.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-03-28T15:40:16.864 INFO:teuthology.orchestra.run.smithi023.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-28T15:40:16.889 INFO:teuthology.orchestra.run.smithi023.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 712 kB/s | 309 kB 00:00 2024-03-28T15:40:17.265 INFO:teuthology.orchestra.run.smithi023.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 65 kB/s | 26 kB 00:00 2024-03-28T15:40:17.268 INFO:teuthology.orchestra.run.smithi117.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-28T15:40:17.290 INFO:teuthology.orchestra.run.smithi023.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 122 kB/s | 49 kB 00:00 2024-03-28T15:40:17.432 INFO:teuthology.orchestra.run.smithi023.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-28T15:40:17.635 INFO:teuthology.orchestra.run.smithi117.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 2.4 MB/s | 890 kB 00:00 2024-03-28T15:40:17.710 INFO:teuthology.orchestra.run.smithi117.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.9 MB/s | 145 kB 00:00 2024-03-28T15:40:17.794 INFO:teuthology.orchestra.run.smithi117.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 7.7 MB/s | 5.4 MB 00:00 2024-03-28T15:40:17.874 INFO:teuthology.orchestra.run.smithi023.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 115 kB/s | 67 kB 00:00 2024-03-28T15:40:17.899 INFO:teuthology.orchestra.run.smithi023.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 248 kB/s | 151 kB 00:00 2024-03-28T15:40:17.933 INFO:teuthology.orchestra.run.smithi023.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.6 MB/s | 838 kB 00:00 2024-03-28T15:40:17.961 INFO:teuthology.orchestra.run.smithi117.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 9.0 MB/s | 1.5 MB 00:00 2024-03-28T15:40:18.092 INFO:teuthology.orchestra.run.smithi023.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-03-28T15:40:18.117 INFO:teuthology.orchestra.run.smithi023.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 200 kB/s | 43 kB 00:00 2024-03-28T15:40:18.150 INFO:teuthology.orchestra.run.smithi023.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 148 kB/s | 32 kB 00:00 2024-03-28T15:40:18.209 INFO:teuthology.orchestra.run.smithi023.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-28T15:40:18.259 INFO:teuthology.orchestra.run.smithi023.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.2 MB/s | 172 kB 00:00 2024-03-28T15:40:18.278 INFO:teuthology.orchestra.run.smithi117.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 4.1 MB/s | 2.3 MB 00:00 2024-03-28T15:40:18.393 INFO:teuthology.orchestra.run.smithi023.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.4 MB/s | 358 kB 00:00 2024-03-28T15:40:18.411 INFO:teuthology.orchestra.run.smithi117.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 10 MB/s | 4.7 MB 00:00 2024-03-28T15:40:18.618 INFO:teuthology.orchestra.run.smithi023.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 741 kB/s | 303 kB 00:00 2024-03-28T15:40:18.702 INFO:teuthology.orchestra.run.smithi023.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 80 kB/s | 35 kB 00:00 2024-03-28T15:40:18.802 INFO:teuthology.orchestra.run.smithi023.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 496 kB/s | 203 kB 00:00 2024-03-28T15:40:19.219 INFO:teuthology.orchestra.run.smithi023.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.4 MB/s | 2.0 MB 00:00 2024-03-28T15:40:19.244 INFO:teuthology.orchestra.run.smithi023.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 266 kB/s | 144 kB 00:00 2024-03-28T15:40:19.269 INFO:teuthology.orchestra.run.smithi023.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 23 kB/s | 11 kB 00:00 2024-03-28T15:40:19.270 INFO:teuthology.orchestra.run.smithi117.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 11 MB/s | 9.9 MB 00:00 2024-03-28T15:40:19.312 INFO:teuthology.orchestra.run.smithi117.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 608 kB/s | 25 kB 00:00 2024-03-28T15:40:20.003 INFO:teuthology.orchestra.run.smithi023.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 23 kB/s | 18 kB 00:00 2024-03-28T15:40:20.028 INFO:teuthology.orchestra.run.smithi023.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 30 kB/s | 23 kB 00:00 2024-03-28T15:40:20.053 INFO:teuthology.orchestra.run.smithi023.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 25 kB/s | 19 kB 00:00 2024-03-28T15:40:21.262 INFO:teuthology.orchestra.run.smithi117.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 4.9 MB/s | 20 MB 00:04 2024-03-28T15:40:21.445 INFO:teuthology.orchestra.run.smithi117.stdout:(11/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 5.3 MB/s | 17 MB 00:03 2024-03-28T15:40:21.470 INFO:teuthology.orchestra.run.smithi117.stdout:(12/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 1.0 MB/s | 220 kB 00:00 2024-03-28T15:40:21.496 INFO:teuthology.orchestra.run.smithi117.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 540 kB/s | 27 kB 00:00 2024-03-28T15:40:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 625 kB/s | 31 kB 00:00 2024-03-28T15:40:21.580 INFO:teuthology.orchestra.run.smithi117.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 2.7 MB/s | 163 kB 00:00 2024-03-28T15:40:21.605 INFO:teuthology.orchestra.run.smithi117.stdout:(16/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 6.7 MB/s | 746 kB 00:00 2024-03-28T15:40:21.638 INFO:teuthology.orchestra.run.smithi117.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 127 kB 00:00 2024-03-28T15:40:21.705 INFO:teuthology.orchestra.run.smithi117.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 4.9 MB/s | 501 kB 00:00 2024-03-28T15:40:21.764 INFO:teuthology.orchestra.run.smithi117.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 777 kB/s | 45 kB 00:00 2024-03-28T15:40:21.831 INFO:teuthology.orchestra.run.smithi117.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.9 MB/s | 131 kB 00:00 2024-03-28T15:40:21.897 INFO:teuthology.orchestra.run.smithi117.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.4 MB/s | 162 kB 00:00 2024-03-28T15:40:22.006 INFO:teuthology.orchestra.run.smithi117.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 321 kB 00:00 2024-03-28T15:40:22.037 INFO:teuthology.orchestra.run.smithi023.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 9.7 kB/s | 20 kB 00:02 2024-03-28T15:40:22.072 INFO:teuthology.orchestra.run.smithi120.stdout: Preparing : 1/1 2024-03-28T15:40:22.081 INFO:teuthology.orchestra.run.smithi117.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 298 kB 00:00 2024-03-28T15:40:22.095 INFO:teuthology.orchestra.run.smithi023.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 512 kB/s | 1.0 MB 00:02 2024-03-28T15:40:22.112 INFO:teuthology.orchestra.run.smithi023.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 22 kB/s | 46 kB 00:02 2024-03-28T15:40:22.148 INFO:teuthology.orchestra.run.smithi117.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 99 kB 00:00 2024-03-28T15:40:22.187 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:40:22.207 INFO:teuthology.orchestra.run.smithi117.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-28T15:40:22.212 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:40:22.243 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:22.252 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:22.293 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:40:22.346 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:40:22.380 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:40:22.414 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:40:22.456 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:40:22.554 INFO:teuthology.orchestra.run.smithi023.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 154 kB/s | 79 kB 00:00 2024-03-28T15:40:22.588 INFO:teuthology.orchestra.run.smithi023.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 121 kB/s | 58 kB 00:00 2024-03-28T15:40:22.646 INFO:teuthology.orchestra.run.smithi023.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 511 kB/s | 272 kB 00:00 2024-03-28T15:40:22.666 INFO:teuthology.orchestra.run.smithi117.stdout:(26/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 14 MB/s | 48 MB 00:03 2024-03-28T15:40:22.741 INFO:teuthology.orchestra.run.smithi117.stdout:(27/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 4.7 MB/s | 5.1 MB 00:01 2024-03-28T15:40:22.766 INFO:teuthology.orchestra.run.smithi117.stdout:(28/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.7 MB/s | 170 kB 00:00 2024-03-28T15:40:22.776 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:40:22.792 INFO:teuthology.orchestra.run.smithi117.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2145.g 678 kB/s | 34 kB 00:00 2024-03-28T15:40:22.817 INFO:teuthology.orchestra.run.smithi117.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-28T15:40:22.841 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:22.875 INFO:teuthology.orchestra.run.smithi117.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.6 MB/s | 3.1 MB 00:00 2024-03-28T15:40:22.951 INFO:teuthology.orchestra.run.smithi117.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 3.2 MB/s | 248 kB 00:00 2024-03-28T15:40:22.962 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:22.988 INFO:teuthology.orchestra.run.smithi023.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 38 kB/s | 16 kB 00:00 2024-03-28T15:40:22.992 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:23.018 INFO:teuthology.orchestra.run.smithi117.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 741 kB/s | 49 kB 00:00 2024-03-28T15:40:23.021 INFO:teuthology.orchestra.run.smithi023.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 207 kB/s | 90 kB 00:00 2024-03-28T15:40:23.039 INFO:teuthology.orchestra.run.smithi023.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 72 kB/s | 28 kB 00:00 2024-03-28T15:40:23.059 INFO:teuthology.orchestra.run.smithi117.stdout:(34/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 6.7 MB/s | 1.8 MB 00:00 2024-03-28T15:40:23.083 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:23.085 INFO:teuthology.orchestra.run.smithi117.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 246 kB/s | 16 kB 00:00 2024-03-28T15:40:23.113 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:40:23.126 INFO:teuthology.orchestra.run.smithi117.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 3.9 MB/s | 264 kB 00:00 2024-03-28T15:40:23.151 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:40:23.180 INFO:teuthology.orchestra.run.smithi023.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 160 kB/s | 31 kB 00:00 2024-03-28T15:40:23.185 INFO:teuthology.orchestra.run.smithi117.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 313 kB/s | 18 kB 00:00 2024-03-28T15:40:23.231 INFO:teuthology.orchestra.run.smithi023.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 903 kB/s | 188 kB 00:00 2024-03-28T15:40:23.235 INFO:teuthology.orchestra.run.smithi117.stdout:(38/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.9 MB/s | 748 kB 00:00 2024-03-28T15:40:23.273 INFO:teuthology.orchestra.run.smithi023.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-03-28T15:40:23.323 INFO:teuthology.orchestra.run.smithi023.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 251 kB/s | 36 kB 00:00 2024-03-28T15:40:23.325 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:40:23.378 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:40:23.386 INFO:teuthology.orchestra.run.smithi117.stdout:(39/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-28T15:40:23.397 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:23.408 INFO:teuthology.orchestra.run.smithi023.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 288 kB/s | 51 kB 00:00 2024-03-28T15:40:23.509 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:23.532 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:23.585 INFO:teuthology.orchestra.run.smithi023.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 735 kB/s | 230 kB 00:00 2024-03-28T15:40:23.596 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:23.644 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:40:23.667 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:23.683 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:23.686 INFO:teuthology.orchestra.run.smithi117.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.3 MB/s | 662 kB 00:00 2024-03-28T15:40:23.705 INFO:teuthology.orchestra.run.smithi117.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 361 kB/s | 106 kB 00:00 2024-03-28T15:40:23.756 INFO:teuthology.orchestra.run.smithi117.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.4 MB/s | 307 kB 00:00 2024-03-28T15:40:23.789 INFO:teuthology.orchestra.run.smithi117.stdout:(43/174): bison-3.7.4-5.el9.x86_64.rpm 1.7 MB/s | 920 kB 00:00 2024-03-28T15:40:23.814 INFO:teuthology.orchestra.run.smithi117.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 291 kB/s | 32 kB 00:00 2024-03-28T15:40:23.875 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:23.890 INFO:teuthology.orchestra.run.smithi023.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 158 kB/s | 90 kB 00:00 2024-03-28T15:40:23.906 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:23.938 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:40:23.941 INFO:teuthology.orchestra.run.smithi023.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.2 MB/s | 427 kB 00:00 2024-03-28T15:40:23.970 INFO:teuthology.orchestra.run.smithi023.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 52 kB/s | 20 kB 00:00 2024-03-28T15:40:24.023 INFO:teuthology.orchestra.run.smithi117.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-03-28T15:40:24.041 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:40:24.049 INFO:teuthology.orchestra.run.smithi117.stdout:(46/174): flexiblas-openblas-openmp-3.0.4-8.el9 73 kB/s | 17 kB 00:00 2024-03-28T15:40:24.074 INFO:teuthology.orchestra.run.smithi117.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 39 kB/s | 10 kB 00:00 2024-03-28T15:40:24.108 INFO:teuthology.orchestra.run.smithi117.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-28T15:40:24.141 INFO:teuthology.orchestra.run.smithi117.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 666 kB/s | 44 kB 00:00 2024-03-28T15:40:24.149 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:24.183 INFO:teuthology.orchestra.run.smithi117.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-28T15:40:24.210 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:24.233 INFO:teuthology.orchestra.run.smithi117.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 4.5 MB/s | 578 kB 00:00 2024-03-28T15:40:24.286 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:24.292 INFO:teuthology.orchestra.run.smithi117.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 310 kB/s | 46 kB 00:00 2024-03-28T15:40:24.317 INFO:teuthology.orchestra.run.smithi117.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-03-28T15:40:24.342 INFO:teuthology.orchestra.run.smithi117.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-28T15:40:24.344 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:24.376 INFO:teuthology.orchestra.run.smithi117.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 588 kB/s | 34 kB 00:00 2024-03-28T15:40:24.401 INFO:teuthology.orchestra.run.smithi117.stdout:(56/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-03-28T15:40:24.579 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:24.650 INFO:teuthology.orchestra.run.smithi023.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 253 kB/s | 192 kB 00:00 2024-03-28T15:40:24.709 INFO:teuthology.orchestra.run.smithi117.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-03-28T15:40:24.742 INFO:teuthology.orchestra.run.smithi023.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-28T15:40:24.743 INFO:teuthology.orchestra.run.smithi117.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-03-28T15:40:24.793 INFO:teuthology.orchestra.run.smithi117.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 198 kB/s | 78 kB 00:00 2024-03-28T15:40:24.800 INFO:teuthology.orchestra.run.smithi023.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-28T15:40:24.907 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:40:25.001 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:40:25.050 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:40:25.067 INFO:teuthology.orchestra.run.smithi023.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 7.5 MB/s | 3.1 MB 00:00 2024-03-28T15:40:25.076 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:40:25.085 INFO:teuthology.orchestra.run.smithi117.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-28T15:40:25.110 INFO:teuthology.orchestra.run.smithi117.stdout:(61/174): python3-jsonpointer-2.0-4.el9.noarch. 57 kB/s | 18 kB 00:00 2024-03-28T15:40:25.111 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:40:25.136 INFO:teuthology.orchestra.run.smithi117.stdout:(62/174): python3-jsonpatch-1.21-16.el9.noarch. 72 kB/s | 24 kB 00:00 2024-03-28T15:40:25.161 INFO:teuthology.orchestra.run.smithi117.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T15:40:25.186 INFO:teuthology.orchestra.run.smithi117.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 419 kB/s | 32 kB 00:00 2024-03-28T15:40:25.193 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:40:25.237 INFO:teuthology.orchestra.run.smithi117.stdout:(65/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.8 MB/s | 217 kB 00:00 2024-03-28T15:40:25.248 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:40:25.287 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:40:25.351 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:40:25.386 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:40:25.430 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:40:25.462 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:40:25.470 INFO:teuthology.orchestra.run.smithi117.stdout:(66/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-28T15:40:25.490 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:40:25.495 INFO:teuthology.orchestra.run.smithi117.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 566 kB/s | 174 kB 00:00 2024-03-28T15:40:25.512 INFO:teuthology.orchestra.run.smithi117.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 250 kB/s | 69 kB 00:00 2024-03-28T15:40:25.529 INFO:teuthology.orchestra.run.smithi117.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 708 kB/s | 41 kB 00:00 2024-03-28T15:40:25.554 INFO:teuthology.orchestra.run.smithi117.stdout:(70/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-28T15:40:25.559 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:40:25.588 INFO:teuthology.orchestra.run.smithi117.stdout:(71/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-28T15:40:25.595 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:40:25.622 INFO:teuthology.orchestra.run.smithi117.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.2 MB/s | 211 kB 00:00 2024-03-28T15:40:25.638 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:40:25.647 INFO:teuthology.orchestra.run.smithi117.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 519 kB/s | 47 kB 00:00 2024-03-28T15:40:25.668 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:40:25.674 INFO:teuthology.orchestra.run.smithi117.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-28T15:40:25.735 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:40:25.782 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:40:25.826 INFO:teuthology.orchestra.run.smithi023.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 2.2 MB/s | 2.3 MB 00:01 2024-03-28T15:40:25.856 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:40:25.889 INFO:teuthology.orchestra.run.smithi117.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 160 kB/s | 39 kB 00:00 2024-03-28T15:40:25.893 INFO:teuthology.orchestra.run.smithi023.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.2 MB/s | 1.8 MB 00:00 2024-03-28T15:40:25.914 INFO:teuthology.orchestra.run.smithi117.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 170 kB/s | 41 kB 00:00 2024-03-28T15:40:25.943 INFO:teuthology.orchestra.run.smithi023.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 361 kB/s | 42 kB 00:00 2024-03-28T15:40:25.990 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:40:26.010 INFO:teuthology.orchestra.run.smithi023.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 2.1 MB/s | 248 kB 00:00 2024-03-28T15:40:26.035 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:40:26.060 INFO:teuthology.orchestra.run.smithi023.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 610 kB/s | 71 kB 00:00 2024-03-28T15:40:26.081 INFO:teuthology.orchestra.run.smithi117.stdout:(77/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-03-28T15:40:26.110 INFO:teuthology.orchestra.run.smithi023.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 363 kB/s | 36 kB 00:00 2024-03-28T15:40:26.169 INFO:teuthology.orchestra.run.smithi023.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-28T15:40:26.219 INFO:teuthology.orchestra.run.smithi023.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-28T15:40:26.240 INFO:teuthology.orchestra.run.smithi117.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-28T15:40:26.265 INFO:teuthology.orchestra.run.smithi117.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 162 kB/s | 56 kB 00:00 2024-03-28T15:40:26.269 INFO:teuthology.orchestra.run.smithi023.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 229 kB/s | 23 kB 00:00 2024-03-28T15:40:26.290 INFO:teuthology.orchestra.run.smithi117.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 300 kB/s | 63 kB 00:00 2024-03-28T15:40:26.315 INFO:teuthology.orchestra.run.smithi117.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 882 kB/s | 44 kB 00:00 2024-03-28T15:40:26.328 INFO:teuthology.orchestra.run.smithi023.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 108 kB 00:00 2024-03-28T15:40:26.340 INFO:teuthology.orchestra.run.smithi117.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-28T15:40:26.366 INFO:teuthology.orchestra.run.smithi117.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-28T15:40:26.378 INFO:teuthology.orchestra.run.smithi023.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-28T15:40:26.391 INFO:teuthology.orchestra.run.smithi117.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-28T15:40:26.425 INFO:teuthology.orchestra.run.smithi117.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-28T15:40:26.437 INFO:teuthology.orchestra.run.smithi023.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-28T15:40:26.450 INFO:teuthology.orchestra.run.smithi117.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-28T15:40:26.475 INFO:teuthology.orchestra.run.smithi117.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-28T15:40:26.489 INFO:teuthology.orchestra.run.smithi023.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 588 kB/s | 64 kB 00:00 2024-03-28T15:40:26.537 INFO:teuthology.orchestra.run.smithi023.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-28T15:40:26.548 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:40:26.621 INFO:teuthology.orchestra.run.smithi023.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.1 MB/s | 1.2 MB 00:00 2024-03-28T15:40:26.621 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:40:26.650 INFO:teuthology.orchestra.run.smithi117.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-28T15:40:26.675 INFO:teuthology.orchestra.run.smithi117.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 833 kB/s | 187 kB 00:00 2024-03-28T15:40:26.693 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:40:26.709 INFO:teuthology.orchestra.run.smithi117.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-28T15:40:26.734 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:26.738 INFO:teuthology.orchestra.run.smithi023.stdout:(168/174): librados2-19.0.0-2145.g3be9a5c9.el9. 1.2 MB/s | 3.4 MB 00:02 2024-03-28T15:40:26.743 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:26.774 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:40:26.815 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:40:26.818 INFO:teuthology.orchestra.run.smithi117.stdout:(91/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-28T15:40:26.853 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:40:26.880 INFO:teuthology.orchestra.run.smithi023.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.3 MB/s | 2.2 MB 00:00 2024-03-28T15:40:26.883 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:40:26.960 INFO:teuthology.orchestra.run.smithi117.stdout:(92/174): elfutils-libelf-devel-0.190-2.el9.x86 76 kB/s | 21 kB 00:00 2024-03-28T15:40:26.985 INFO:teuthology.orchestra.run.smithi117.stdout:(93/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 92 kB/s | 25 kB 00:00 2024-03-28T15:40:27.005 INFO:teuthology.orchestra.run.smithi023.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 249 kB/s | 31 kB 00:00 2024-03-28T15:40:27.113 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:40:27.139 INFO:teuthology.orchestra.run.smithi023.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.4 MB/s | 2.3 MB 00:00 2024-03-28T15:40:27.276 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:40:27.345 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:40:27.347 INFO:teuthology.orchestra.run.smithi023.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 357 kB/s | 74 kB 00:00 2024-03-28T15:40:27.389 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:40:27.414 INFO:teuthology.orchestra.run.smithi023.stdout:(173/174): util-linux-core-2.37.4-18.el9.x86_64 688 kB/s | 465 kB 00:00 2024-03-28T15:40:27.517 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:40:27.552 INFO:teuthology.orchestra.run.smithi117.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 1.6 MB/s | 911 kB 00:00 2024-03-28T15:40:27.558 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:40:27.620 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:40:27.677 INFO:teuthology.orchestra.run.smithi117.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-03-28T15:40:27.864 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:40:27.893 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:40:27.929 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:40:27.976 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:40:28.013 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:40:28.055 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:40:28.097 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:40:28.230 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:40:28.277 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:40:28.299 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:40:28.341 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:40:28.434 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:40:28.466 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:40:28.481 INFO:teuthology.orchestra.run.smithi023.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 745 kB/s | 1.1 MB 00:01 2024-03-28T15:40:28.488 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:40:28.488 INFO:teuthology.orchestra.run.smithi023.stdout:Total 7.3 MB/s | 225 MB 00:30 2024-03-28T15:40:28.506 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:40:28.532 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:40:28.557 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:40:28.628 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:40:28.757 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:40:28.864 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:40:28.908 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:40:28.956 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:40:29.011 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:40:29.086 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction check 2024-03-28T15:40:29.136 INFO:teuthology.orchestra.run.smithi117.stdout:(96/174): postgresql-13.14-1.el9.x86_64.rpm 747 kB/s | 1.6 MB 00:02 2024-03-28T15:40:29.193 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:40:29.203 INFO:teuthology.orchestra.run.smithi117.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 1.7 MB/s | 4.1 MB 00:02 2024-03-28T15:40:29.245 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:40:29.263 INFO:teuthology.orchestra.run.smithi117.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-28T15:40:29.330 INFO:teuthology.orchestra.run.smithi117.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-28T15:40:29.372 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:40:29.384 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction check succeeded. 2024-03-28T15:40:29.384 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction test 2024-03-28T15:40:29.397 INFO:teuthology.orchestra.run.smithi117.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-03-28T15:40:29.410 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:40:29.442 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:40:29.472 INFO:teuthology.orchestra.run.smithi117.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 253 kB/s | 36 kB 00:00 2024-03-28T15:40:29.473 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:40:29.501 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:40:29.536 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:40:29.539 INFO:teuthology.orchestra.run.smithi117.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 334 kB/s | 47 kB 00:00 2024-03-28T15:40:29.575 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:40:29.598 INFO:teuthology.orchestra.run.smithi117.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 748 kB/s | 94 kB 00:00 2024-03-28T15:40:29.606 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:40:29.637 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:40:29.712 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:40:29.747 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:40:29.781 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:40:29.822 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:40:29.857 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:40:29.875 INFO:teuthology.orchestra.run.smithi117.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-03-28T15:40:29.945 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:40:30.035 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:40:30.081 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:40:30.099 INFO:teuthology.orchestra.run.smithi117.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-03-28T15:40:30.111 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:40:30.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:40:30.255 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:40:30.312 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:40:30.591 INFO:teuthology.orchestra.run.smithi117.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 765 kB/s | 548 kB 00:00 2024-03-28T15:40:30.699 INFO:teuthology.orchestra.run.smithi117.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 180 kB/s | 19 kB 00:00 2024-03-28T15:40:30.799 INFO:teuthology.orchestra.run.smithi117.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 442 kB/s | 309 kB 00:00 2024-03-28T15:40:30.900 INFO:teuthology.orchestra.run.smithi117.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-28T15:40:31.008 INFO:teuthology.orchestra.run.smithi117.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-28T15:40:31.032 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:40:31.077 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:40:31.092 INFO:teuthology.orchestra.run.smithi117.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-28T15:40:31.111 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:40:31.117 INFO:teuthology.orchestra.run.smithi117.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 621 kB/s | 67 kB 00:00 2024-03-28T15:40:31.158 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:40:31.192 INFO:teuthology.orchestra.run.smithi117.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-03-28T15:40:31.237 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:40:31.279 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:40:31.310 INFO:teuthology.orchestra.run.smithi117.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-03-28T15:40:31.376 INFO:teuthology.orchestra.run.smithi117.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-03-28T15:40:31.418 INFO:teuthology.orchestra.run.smithi117.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-03-28T15:40:31.457 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:40:31.477 INFO:teuthology.orchestra.run.smithi117.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-28T15:40:31.478 INFO:teuthology.orchestra.run.smithi023.stdout:Transaction test succeeded. 2024-03-28T15:40:31.478 INFO:teuthology.orchestra.run.smithi023.stdout:Running transaction 2024-03-28T15:40:31.485 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:40:31.519 INFO:teuthology.orchestra.run.smithi117.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-28T15:40:31.562 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:40:31.586 INFO:teuthology.orchestra.run.smithi117.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-28T15:40:31.620 INFO:teuthology.orchestra.run.smithi117.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-28T15:40:31.703 INFO:teuthology.orchestra.run.smithi117.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-03-28T15:40:31.728 INFO:teuthology.orchestra.run.smithi117.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-03-28T15:40:31.812 INFO:teuthology.orchestra.run.smithi117.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-28T15:40:31.870 INFO:teuthology.orchestra.run.smithi117.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-28T15:40:31.921 INFO:teuthology.orchestra.run.smithi117.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-28T15:40:31.963 INFO:teuthology.orchestra.run.smithi117.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-28T15:40:32.014 INFO:teuthology.orchestra.run.smithi117.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-03-28T15:40:32.055 INFO:teuthology.orchestra.run.smithi117.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-03-28T15:40:32.105 INFO:teuthology.orchestra.run.smithi117.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-03-28T15:40:32.147 INFO:teuthology.orchestra.run.smithi117.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-28T15:40:32.196 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:40:32.239 INFO:teuthology.orchestra.run.smithi117.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-03-28T15:40:32.281 INFO:teuthology.orchestra.run.smithi117.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-28T15:40:32.306 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:40:32.332 INFO:teuthology.orchestra.run.smithi117.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-03-28T15:40:32.381 INFO:teuthology.orchestra.run.smithi117.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-28T15:40:32.432 INFO:teuthology.orchestra.run.smithi117.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-28T15:40:32.482 INFO:teuthology.orchestra.run.smithi117.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-28T15:40:32.532 INFO:teuthology.orchestra.run.smithi117.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 896 kB/s | 90 kB 00:00 2024-03-28T15:40:32.591 INFO:teuthology.orchestra.run.smithi117.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-03-28T15:40:32.633 INFO:teuthology.orchestra.run.smithi117.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 307 kB/s | 31 kB 00:00 2024-03-28T15:40:32.700 INFO:teuthology.orchestra.run.smithi117.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-28T15:40:32.733 INFO:teuthology.orchestra.run.smithi117.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-03-28T15:40:32.800 INFO:teuthology.orchestra.run.smithi117.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-28T15:40:32.834 INFO:teuthology.orchestra.run.smithi117.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-28T15:40:32.909 INFO:teuthology.orchestra.run.smithi117.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-28T15:40:32.934 INFO:teuthology.orchestra.run.smithi117.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 895 kB/s | 90 kB 00:00 2024-03-28T15:40:33.027 INFO:teuthology.orchestra.run.smithi117.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-28T15:40:33.052 INFO:teuthology.orchestra.run.smithi117.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-28T15:40:33.135 INFO:teuthology.orchestra.run.smithi117.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-28T15:40:33.186 INFO:teuthology.orchestra.run.smithi117.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-28T15:40:33.511 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:40:33.596 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:40:33.619 INFO:teuthology.orchestra.run.smithi117.stdout:(150/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.0 MB/s | 3.4 MB 00:00 2024-03-28T15:40:33.627 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:40:33.658 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:40:33.686 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:40:33.744 INFO:teuthology.orchestra.run.smithi117.stdout:(151/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.9 MB/s | 3.1 MB 00:00 2024-03-28T15:40:33.937 INFO:teuthology.orchestra.run.smithi117.stdout:(152/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.0 MB/s | 321 kB 00:00 2024-03-28T15:40:33.997 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:40:34.012 INFO:teuthology.orchestra.run.smithi117.stdout:(153/174): postgresql-server-13.14-1.el9.x86_64 934 kB/s | 5.8 MB 00:06 2024-03-28T15:40:34.024 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:34.114 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:34.146 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:40:34.162 INFO:teuthology.orchestra.run.smithi117.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 281 kB/s | 42 kB 00:00 2024-03-28T15:40:34.181 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:40:34.304 INFO:teuthology.orchestra.run.smithi117.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-28T15:40:34.371 INFO:teuthology.orchestra.run.smithi117.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.0 MB/s | 71 kB 00:00 2024-03-28T15:40:34.430 INFO:teuthology.orchestra.run.smithi117.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 625 kB/s | 36 kB 00:00 2024-03-28T15:40:34.513 INFO:teuthology.orchestra.run.smithi117.stdout:(158/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.0 MB/s | 2.3 MB 00:00 2024-03-28T15:40:34.547 INFO:teuthology.orchestra.run.smithi117.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.6 MB/s | 192 kB 00:00 2024-03-28T15:40:34.580 INFO:teuthology.orchestra.run.smithi117.stdout:(160/174): NetworkManager-libnm-1.47.2-1.el9.x8 2.8 MB/s | 1.8 MB 00:00 2024-03-28T15:40:34.606 INFO:teuthology.orchestra.run.smithi117.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-28T15:40:34.631 INFO:teuthology.orchestra.run.smithi117.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-28T15:40:34.640 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:40:34.656 INFO:teuthology.orchestra.run.smithi117.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-03-28T15:40:34.661 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:34.681 INFO:teuthology.orchestra.run.smithi117.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-03-28T15:40:34.707 INFO:teuthology.orchestra.run.smithi117.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-28T15:40:34.714 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:34.723 INFO:teuthology.orchestra.run.smithi117.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 951 kB/s | 64 kB 00:00 2024-03-28T15:40:34.737 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:40:34.740 INFO:teuthology.orchestra.run.smithi117.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 489 kB/s | 29 kB 00:00 2024-03-28T15:40:34.999 INFO:teuthology.orchestra.run.smithi117.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.1 MB/s | 1.2 MB 00:00 2024-03-28T15:40:35.149 INFO:teuthology.orchestra.run.smithi117.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.1 MB/s | 465 kB 00:00 2024-03-28T15:40:35.200 INFO:teuthology.orchestra.run.smithi117.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 623 kB/s | 31 kB 00:00 2024-03-28T15:40:35.308 INFO:teuthology.orchestra.run.smithi117.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.0 MB/s | 2.3 MB 00:00 2024-03-28T15:40:35.350 INFO:teuthology.orchestra.run.smithi117.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 3.4 MB/s | 2.2 MB 00:00 2024-03-28T15:40:35.500 INFO:teuthology.orchestra.run.smithi117.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.6 MB/s | 1.1 MB 00:00 2024-03-28T15:40:35.617 INFO:teuthology.orchestra.run.smithi117.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 241 kB/s | 74 kB 00:00 2024-03-28T15:40:35.622 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:40:35.622 INFO:teuthology.orchestra.run.smithi117.stdout:Total 12 MB/s | 225 MB 00:19 2024-03-28T15:40:36.182 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-28T15:40:36.458 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-28T15:40:36.458 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-28T15:40:36.925 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:36.951 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:37.484 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:37.530 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:40:37.594 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:40:37.594 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:40:37.594 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:40:37.594 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:37.614 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:40:38.192 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-28T15:40:38.193 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-28T15:40:38.647 INFO:teuthology.orchestra.run.smithi023.stdout: Preparing : 1/1 2024-03-28T15:40:38.769 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:40:38.804 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:40:38.852 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:38.863 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:38.911 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:40:38.966 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:40:39.007 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:40:39.032 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:40:39.086 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:40:39.428 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:40:39.474 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:39.606 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:39.634 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:39.718 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:39.754 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:40:39.798 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:40:39.990 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:40:40.037 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:40:40.064 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:40.169 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:40.198 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:40.263 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:40.302 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:40:40.333 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:40.348 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:40.517 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:40.552 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:40.596 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:40:40.770 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:40:40.890 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:40.950 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:41.043 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:41.093 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:41.287 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:41.402 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:40:41.513 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:40:41.548 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:40:41.576 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:40:41.620 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:40:41.695 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:40:41.747 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:40:41.793 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:40:41.859 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:40:41.893 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:40:41.936 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:40:41.969 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:40:41.996 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:40:42.067 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:40:42.110 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:40:42.159 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:40:42.190 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:40:42.253 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:40:42.305 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:40:42.380 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:40:42.526 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:40:42.599 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:40:43.153 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:40:43.237 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:40:43.315 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:40:43.373 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:43.383 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:43.426 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:40:43.464 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:40:43.501 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:40:43.539 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:40:43.979 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:40:44.307 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:40:44.400 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:40:44.444 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:40:44.593 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:40:44.638 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:40:44.694 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:40:44.768 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:40:44.813 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:40:44.850 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:40:44.898 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:40:44.933 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:40:44.976 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:40:45.018 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:40:45.158 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:40:45.198 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:40:45.228 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:40:45.241 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-28T15:40:45.279 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:40:45.353 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-28T15:40:45.376 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:40:45.387 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-28T15:40:45.412 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:40:45.427 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:45.436 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-28T15:40:45.452 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:40:45.477 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:40:45.484 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-28T15:40:45.502 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:40:45.538 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-28T15:40:45.572 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-28T15:40:45.580 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:40:45.606 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-28T15:40:45.648 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-28T15:40:45.707 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:40:45.813 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:40:45.860 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:40:45.922 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:40:45.969 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-28T15:40:45.974 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:40:46.024 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:46.144 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:40:46.166 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:46.167 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:40:46.214 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:40:46.257 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:40:46.287 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:40:46.329 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-28T15:40:46.358 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:40:46.396 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:40:46.428 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:40:46.458 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:40:46.486 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:40:46.500 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-28T15:40:46.521 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:40:46.560 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-28T15:40:46.560 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:40:46.588 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:46.591 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:40:46.622 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:40:46.691 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:40:46.699 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:40:46.714 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:46.732 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:40:46.761 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-28T15:40:46.765 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:40:46.801 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-28T15:40:46.806 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:40:46.824 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:46.839 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-28T15:40:46.841 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:40:46.933 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:40:47.006 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:47.023 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:40:47.038 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-28T15:40:47.061 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-28T15:40:47.072 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:40:47.103 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:40:47.151 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:40:47.169 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-28T15:40:47.252 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:40:47.272 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:47.306 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:40:47.316 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-28T15:40:47.375 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:47.450 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:47.665 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-28T15:40:47.771 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-28T15:40:47.859 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-28T15:40:47.888 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-28T15:40:47.915 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-28T15:40:47.950 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-28T15:40:48.024 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-28T15:40:48.069 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:40:48.079 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-28T15:40:48.117 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-28T15:40:48.127 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:40:48.162 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:40:48.190 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-28T15:40:48.210 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:40:48.233 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-28T15:40:48.276 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-28T15:40:48.300 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:40:48.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-28T15:40:48.336 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-28T15:40:48.337 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:40:48.405 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-28T15:40:48.441 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-28T15:40:48.485 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-28T15:40:48.513 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-28T15:40:48.522 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:40:48.551 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:40:48.593 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-28T15:40:48.632 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:40:48.636 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-28T15:40:48.711 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-28T15:40:48.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-28T15:40:48.897 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-28T15:40:49.321 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:40:49.419 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-28T15:40:49.436 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:40:49.500 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-28T15:40:49.583 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-28T15:40:49.605 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:40:49.605 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /sys 2024-03-28T15:40:49.605 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /proc 2024-03-28T15:40:49.605 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /mnt 2024-03-28T15:40:49.606 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /var/tmp 2024-03-28T15:40:49.606 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /home 2024-03-28T15:40:49.606 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /root 2024-03-28T15:40:49.606 INFO:teuthology.orchestra.run.smithi120.stdout:skipping the directory /tmp 2024-03-28T15:40:49.606 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:49.629 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:49.638 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-28T15:40:49.669 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-28T15:40:49.718 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-28T15:40:49.756 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-28T15:40:49.767 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:40:49.782 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:40:49.783 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:49.783 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:40:49.783 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:49.785 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-28T15:40:50.027 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-28T15:40:50.196 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-28T15:40:50.263 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-28T15:40:50.298 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-28T15:40:50.444 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-28T15:40:50.485 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-28T15:40:50.549 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-28T15:40:50.626 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-28T15:40:50.653 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-28T15:40:50.738 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:40:50.773 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-28T15:40:50.826 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:40:50.857 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:40:50.888 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:40:50.916 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:40:50.922 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-28T15:40:50.956 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-28T15:40:51.000 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-28T15:40:51.041 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-28T15:40:51.176 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-28T15:40:51.238 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:40:51.263 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-28T15:40:51.270 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:51.279 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:40:51.301 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-28T15:40:51.304 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:40:51.304 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:51.305 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:40:51.305 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:40:51.305 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:40:51.305 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:51.344 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-28T15:40:51.362 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:51.392 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:40:51.429 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:40:51.437 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-28T15:40:51.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-28T15:40:51.508 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-28T15:40:51.521 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:40:51.542 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:40:51.544 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:51.567 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-28T15:40:51.638 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-28T15:40:51.768 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-28T15:40:51.868 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-28T15:40:51.909 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-28T15:40:51.913 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:40:51.940 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:51.956 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-28T15:40:51.989 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:40:52.011 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:52.012 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-28T15:40:52.042 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:40:52.108 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:40:52.196 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-28T15:40:52.255 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-28T15:40:52.393 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-28T15:40:52.435 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-28T15:40:52.468 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-28T15:40:52.498 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-28T15:40:52.500 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:40:52.527 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-28T15:40:52.531 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:40:52.555 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:40:52.561 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-28T15:40:52.600 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-28T15:40:52.607 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:40:52.632 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-28T15:40:52.638 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:40:52.662 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-28T15:40:52.739 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-28T15:40:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-28T15:40:52.805 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-28T15:40:52.847 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-28T15:40:52.871 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:40:52.881 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-28T15:40:52.906 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:40:52.970 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-28T15:40:53.061 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-28T15:40:53.104 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-28T15:40:53.135 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-28T15:40:53.183 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-28T15:40:53.280 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-28T15:40:53.328 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-28T15:40:53.867 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:40:53.889 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:40:53.990 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:40:54.057 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-28T15:40:54.108 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-28T15:40:54.142 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-28T15:40:54.144 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:40:54.167 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:40:54.190 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:54.191 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-28T15:40:54.226 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:40:54.237 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:40:54.273 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:40:54.287 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-28T15:40:54.313 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:54.327 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-28T15:40:54.356 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:54.504 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-28T15:40:54.533 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-28T15:40:54.610 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-28T15:40:54.925 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:40:54.959 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:40:55.041 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:40:55.041 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:40:55.041 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:40:55.041 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:40:55.077 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:40:55.145 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:40:55.178 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:40:55.201 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:55.245 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-28T15:40:55.246 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:40:55.267 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:40:55.267 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:55.267 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:40:55.268 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:55.313 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:40:55.335 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:55.346 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-28T15:40:55.639 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:40:55.661 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:40:55.661 INFO:teuthology.orchestra.run.smithi120.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:40:55.661 INFO:teuthology.orchestra.run.smithi120.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:40:55.661 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:40:55.662 INFO:teuthology.orchestra.run.smithi120.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:40:55.662 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:40:56.558 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-28T15:40:56.641 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-28T15:40:56.672 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-28T15:40:56.703 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-28T15:40:56.731 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-28T15:40:57.044 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-28T15:40:57.069 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:57.151 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-28T15:40:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-28T15:40:57.218 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-28T15:40:57.680 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-28T15:40:57.714 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:57.785 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-28T15:40:57.807 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-28T15:41:00.068 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:00.069 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:41:00.113 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:41:00.117 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:41:00.157 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:41:00.184 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:41:00.365 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:41:00.475 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:41:00.515 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:41:00.546 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:41:00.599 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:41:00.634 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:41:00.649 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-28T15:41:00.659 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:41:00.683 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:41:00.691 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:41:00.713 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:41:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-28T15:41:00.755 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T15:41:00.755 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T15:41:00.755 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:00.766 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:41:00.784 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:41:00.845 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:41:00.879 INFO:teuthology.orchestra.run.smithi120.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:41:01.465 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:01.502 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:01.522 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:02.335 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:41:02.415 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:41:02.456 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:41:02.537 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:41:02.569 INFO:teuthology.orchestra.run.smithi120.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:41:02.594 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:41:02.620 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:41:02.646 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:41:02.646 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:02.735 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:02.764 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:41:02.764 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:02.857 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:02.883 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:41:02.917 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:41:02.942 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:41:02.967 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:41:02.992 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:41:03.031 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:41:03.064 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:41:03.089 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:41:03.115 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:41:03.141 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:03.165 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:41:03.172 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:03.172 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:03.352 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:03.352 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:03.427 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:03.451 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:03.484 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:41:03.511 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:41:03.538 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:41:03.600 INFO:teuthology.orchestra.run.smithi120.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:07.547 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /sys 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /proc 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /mnt 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /var/tmp 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /home 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /root 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout:skipping the directory /tmp 2024-03-28T15:41:07.548 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:07.722 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:41:07.737 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:41:07.737 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:07.737 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:41:07.737 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:41:08.562 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:41:08.563 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:41:08.584 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:41:08.584 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:41:08.584 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:41:08.585 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:41:08.586 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:41:08.587 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:41:08.588 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:41:08.589 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:41:08.590 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:41:08.591 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:41:08.592 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:41:08.593 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:41:08.594 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:41:08.595 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:41:08.596 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:41:08.597 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:41:08.598 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:08.598 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:41:08.598 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:41:08.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:41:08.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:08.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:41:08.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:41:08.599 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:41:09.275 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:41:09.301 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:41:09.301 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:09.301 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:41:09.301 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:41:09.301 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:41:09.302 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:09.534 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:41:09.556 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:41:09.556 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:09.556 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:41:09.556 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:41:09.557 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:41:09.557 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:41:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:41:10.040 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:10.040 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:41:10.040 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:41:10.040 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:41:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:10.136 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:41:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:41:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:41:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:41:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:41:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:41:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:41:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout:Upgraded: 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:41:11.683 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.684 INFO:teuthology.orchestra.run.smithi120.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout:Installed: 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.685 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.686 INFO:teuthology.orchestra.run.smithi120.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:41:11.687 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.688 INFO:teuthology.orchestra.run.smithi120.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:41:11.689 INFO:teuthology.orchestra.run.smithi120.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:41:11.690 INFO:teuthology.orchestra.run.smithi120.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:41:11.691 INFO:teuthology.orchestra.run.smithi120.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:41:11.691 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:41:11.691 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.691 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.691 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:41:11.692 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:41:11.693 INFO:teuthology.orchestra.run.smithi120.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:41:11.694 INFO:teuthology.orchestra.run.smithi120.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:41:11.695 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:41:11.696 INFO:teuthology.orchestra.run.smithi120.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:41:11.697 INFO:teuthology.orchestra.run.smithi120.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:41:11.698 INFO:teuthology.orchestra.run.smithi120.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:41:11.699 INFO:teuthology.orchestra.run.smithi120.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:41:11.699 INFO:teuthology.orchestra.run.smithi120.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:41:11.699 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:41:11.699 INFO:teuthology.orchestra.run.smithi120.stdout:Complete! 2024-03-28T15:41:12.024 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:41:12.050 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:41:12.162 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:41:12.199 DEBUG:teuthology.parallel:result is None 2024-03-28T15:41:12.329 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:41:12.353 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:41:12.376 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:12.420 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:41:12.431 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:41:12.480 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-28T15:41:12.622 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-28T15:41:12.623 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:12.787 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:41:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-28T15:41:12.803 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:12.803 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T15:41:12.803 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:13.404 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:41:13.430 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:41:13.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:41:13.519 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:41:13.519 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:13.519 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:41:13.519 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:13.565 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:13.587 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:13.587 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:13.587 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:41:13.587 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:41:13.587 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:41:13.588 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:13.916 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:41:13.938 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:41:13.938 INFO:teuthology.orchestra.run.smithi023.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:13.939 INFO:teuthology.orchestra.run.smithi023.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:41:13.939 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:41:13.939 INFO:teuthology.orchestra.run.smithi023.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:41:13.939 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:14.308 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:41:14.332 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-28T15:41:14.332 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:14.332 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T15:41:14.333 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:41:14.333 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T15:41:14.333 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:14.557 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:41:14.579 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-28T15:41:14.580 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:14.580 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T15:41:14.580 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:41:14.580 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T15:41:14.580 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:15.025 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T15:41:15.047 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:15.135 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-28T15:41:15.521 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:41:15.552 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-28T15:41:15.584 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-28T15:41:15.643 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-28T15:41:15.666 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:41:15.899 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-28T15:41:15.926 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:41:16.891 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-28T15:41:16.918 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:41:17.018 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-28T15:41:17.172 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-28T15:41:17.203 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:41:17.225 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-28T15:41:17.225 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:17.226 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T15:41:17.226 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:41:17.226 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T15:41:17.226 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:17.270 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:41:17.281 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-28T15:41:17.327 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-28T15:41:18.202 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-28T15:41:18.231 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:41:18.253 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-28T15:41:18.253 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:18.253 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T15:41:18.253 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:41:18.254 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T15:41:18.254 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:18.298 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:41:18.320 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-28T15:41:18.320 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:18.320 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T15:41:18.320 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:18.365 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:18.387 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:18.387 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:18.388 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-28T15:41:18.388 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:41:18.388 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-28T15:41:18.388 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:18.661 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:18.699 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:41:18.703 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T15:41:18.722 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:18.733 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:41:18.760 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:41:18.960 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:41:19.088 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:41:19.134 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:41:19.192 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:41:19.266 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:41:19.301 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:41:19.334 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:41:19.367 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:41:19.396 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:41:19.469 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:41:19.547 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:41:19.586 INFO:teuthology.orchestra.run.smithi023.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:41:20.165 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:20.200 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:20.220 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:21.098 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:41:21.179 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:41:21.220 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:41:21.314 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:41:21.348 INFO:teuthology.orchestra.run.smithi023.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:41:21.365 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:41:21.390 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:41:21.417 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:41:21.417 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:21.497 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:21.527 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:41:21.527 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:21.611 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:21.639 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:41:21.671 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:41:21.696 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:41:21.721 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:41:21.756 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:41:21.802 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:41:21.836 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:41:21.861 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:41:21.886 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:41:21.911 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:21.936 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:41:21.943 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:21.944 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:22.123 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:22.123 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:22.206 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:22.230 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:22.262 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:41:22.290 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:41:22.323 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:41:22.392 INFO:teuthology.orchestra.run.smithi023.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:23.170 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:23.211 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-28T15:41:23.242 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-28T15:41:23.269 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-28T15:41:23.449 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-28T15:41:23.560 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-28T15:41:23.618 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-28T15:41:23.655 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-28T15:41:23.717 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-28T15:41:23.760 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-28T15:41:23.793 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-28T15:41:23.825 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-28T15:41:23.855 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-28T15:41:23.908 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-28T15:41:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-28T15:41:24.011 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-28T15:41:24.600 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:24.643 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:24.662 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-28T15:41:25.478 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-28T15:41:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-28T15:41:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-28T15:41:25.693 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-28T15:41:25.724 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-28T15:41:25.750 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-28T15:41:25.775 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-28T15:41:25.801 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-28T15:41:25.801 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:25.881 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-28T15:41:25.911 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-28T15:41:25.911 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:25.978 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-28T15:41:26.004 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-28T15:41:26.038 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-28T15:41:26.063 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-28T15:41:26.080 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-28T15:41:26.113 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-28T15:41:26.152 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-28T15:41:26.185 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-28T15:41:26.210 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-28T15:41:26.235 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-28T15:41:26.261 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:26.286 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-28T15:41:26.292 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:26.292 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:26.480 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-28T15:41:26.480 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:26.555 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-28T15:41:26.580 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:26.611 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-28T15:41:26.647 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-28T15:41:26.672 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-28T15:41:26.736 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:41:27.209 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:41:27.210 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:41:27.211 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:41:27.228 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:41:27.229 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:41:27.230 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:41:27.231 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:41:27.232 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:41:27.233 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:41:27.234 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:41:27.235 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:41:27.236 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:41:27.237 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:41:27.238 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:41:27.239 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:41:27.240 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout:Upgraded: 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:30.078 INFO:teuthology.orchestra.run.smithi023.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:30.079 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout:Installed: 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.080 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.081 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:41:30.082 INFO:teuthology.orchestra.run.smithi023.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:41:30.083 INFO:teuthology.orchestra.run.smithi023.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:41:30.084 INFO:teuthology.orchestra.run.smithi023.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:41:30.085 INFO:teuthology.orchestra.run.smithi023.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:41:30.086 INFO:teuthology.orchestra.run.smithi023.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:41:30.087 INFO:teuthology.orchestra.run.smithi023.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:41:30.088 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:41:30.089 INFO:teuthology.orchestra.run.smithi023.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:41:30.090 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:41:30.091 INFO:teuthology.orchestra.run.smithi023.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:41:30.092 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:41:30.092 INFO:teuthology.orchestra.run.smithi023.stdout:Complete! 2024-03-28T15:41:30.554 DEBUG:teuthology.parallel:result is None 2024-03-28T15:41:31.542 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-28T15:41:31.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-28T15:41:31.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-28T15:41:31.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-28T15:41:31.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-28T15:41:31.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-28T15:41:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-28T15:41:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-28T15:41:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-28T15:41:31.561 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-28T15:41:31.562 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-28T15:41:31.563 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-28T15:41:31.564 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-28T15:41:31.565 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-28T15:41:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-28T15:41:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-28T15:41:31.568 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-28T15:41:31.569 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-28T15:41:31.570 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-28T15:41:31.571 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-28T15:41:31.572 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-28T15:41:31.573 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-28T15:41:31.574 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-28T15:41:34.512 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-28T15:41:34.512 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: acl-2.3.1-4.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.513 INFO:teuthology.orchestra.run.smithi117.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-28T15:41:34.514 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.515 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.516 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T15:41:34.517 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T15:41:34.518 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.14-1.el9.x86_64 2024-03-28T15:41:34.519 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.520 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T15:41:34.521 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T15:41:34.522 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T15:41:34.523 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-28T15:41:34.524 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-28T15:41:34.525 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T15:41:34.526 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-28T15:41:34.526 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-28T15:41:34.526 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-28T15:41:35.022 DEBUG:teuthology.parallel:result is None 2024-03-28T15:41:35.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:35.243 DEBUG:teuthology.orchestra.run.smithi023:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:41:35.269 INFO:teuthology.orchestra.run.smithi023.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.269 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.269 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:41:35.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:35.414 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:41:35.435 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.435 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.436 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:41:35.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:35.649 DEBUG:teuthology.orchestra.run.smithi120:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T15:41:35.675 INFO:teuthology.orchestra.run.smithi120.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.675 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-28T15:41:35.675 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-28T15:41:35.677 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T15:41:35.677 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:41:35.677 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:41:35.710 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:41:35.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:41:35.739 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:41:35.739 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T15:41:35.772 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T15:41:35.772 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:41:35.772 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:41:35.801 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:41:35.871 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:41:35.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:41:35.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:41:35.964 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:41:35.964 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T15:41:35.993 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T15:41:36.064 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T15:41:36.064 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:41:36.064 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:41:36.096 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:41:36.168 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:41:36.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:41:36.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:41:36.262 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:41:36.262 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T15:41:36.289 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T15:41:36.357 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T15:41:36.357 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:41:36.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:41:36.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:41:36.454 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:41:36.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:41:36.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:41:36.547 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:41:36.547 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T15:41:36.574 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T15:41:36.642 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-28T15:41:36.738 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-28T15:41:36.738 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:36.739 INFO:tasks.cephadm:Cluster fsid is a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:41:36.739 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-28T15:41:36.739 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.23', 'mon.b': '172.21.15.117', 'mon.c': '172.21.15.120'} 2024-03-28T15:41:36.739 INFO:tasks.cephadm:First mon is mon.a on smithi023 2024-03-28T15:41:36.739 INFO:tasks.cephadm:First mgr is x 2024-03-28T15:41:36.739 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-28T15:41:36.739 DEBUG:teuthology.orchestra.run.smithi023:> sudo hostname $(hostname -s) 2024-03-28T15:41:36.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-03-28T15:41:36.796 DEBUG:teuthology.orchestra.run.smithi120:> sudo hostname $(hostname -s) 2024-03-28T15:41:36.823 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-28T15:41:36.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:36.966 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-28T15:41:37.180 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:41:37.182 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-28T15:41:37.182 DEBUG:teuthology.orchestra.run.smithi023:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:38.524 INFO:teuthology.orchestra.run.smithi023.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:41 /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:38.525 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:39.434 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:41 /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:39.434 DEBUG:teuthology.orchestra.run.smithi120:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:40.619 INFO:teuthology.orchestra.run.smithi120.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 28 15:41 /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:40.619 DEBUG:teuthology.orchestra.run.smithi023:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:40.637 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:40.656 DEBUG:teuthology.orchestra.run.smithi120:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-28T15:41:40.691 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-28T15:41:40.692 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:41:40.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:41:40.699 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-28T15:41:41.450 INFO:teuthology.orchestra.run.smithi120.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:41:41.544 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:41:41.565 INFO:teuthology.orchestra.run.smithi023.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:42:08.685 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-03-28T15:42:08.686 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout: "repo_digests": [ 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout: ] 2024-03-28T15:42:14.086 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout: "repo_digests": [ 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T15:42:18.145 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:42:18.169 DEBUG:teuthology.orchestra.run.smithi023:> sudo mkdir -p /etc/ceph 2024-03-28T15:42:18.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-03-28T15:42:18.272 DEBUG:teuthology.orchestra.run.smithi120:> sudo mkdir -p /etc/ceph 2024-03-28T15:42:18.307 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 777 /etc/ceph 2024-03-28T15:42:18.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-03-28T15:42:18.364 DEBUG:teuthology.orchestra.run.smithi120:> sudo chmod 777 /etc/ceph 2024-03-28T15:42:18.395 INFO:tasks.cephadm:Writing seed config... 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-28T15:42:18.396 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-28T15:42:18.397 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-28T15:42:18.398 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-28T15:42:18.399 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-28T15:42:18.400 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-28T15:42:18.400 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-28T15:42:18.400 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:42:18.401 DEBUG:teuthology.orchestra.run.smithi023:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-28T15:42:18.415 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a96af96e-ed19-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-28T15:42:18.416 DEBUG:teuthology.orchestra.run.smithi023:mon.a> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service 2024-03-28T15:42:18.458 DEBUG:teuthology.orchestra.run.smithi023:mgr.x> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x.service 2024-03-28T15:42:18.500 INFO:tasks.cephadm:Bootstrapping... 2024-03-28T15:42:18.501 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.23 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:42:18.697 INFO:teuthology.orchestra.run.smithi023.stdout:-------------------------------------------------------------------------------- 2024-03-28T15:42:18.697 INFO:teuthology.orchestra.run.smithi023.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'a96af96e-ed19-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.23', '--skip-admin-label'] 2024-03-28T15:42:18.721 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:42:18.722 INFO:teuthology.orchestra.run.smithi023.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-28T15:42:18.722 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying podman|docker is present... 2024-03-28T15:42:18.745 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:42:18.745 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying lvm2 is present... 2024-03-28T15:42:18.745 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying time synchronization is in place... 2024-03-28T15:42:18.752 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T15:42:18.752 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T15:42:18.758 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T15:42:18.758 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:42:18.764 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-28T15:42:18.770 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-28T15:42:18.770 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-28T15:42:18.770 INFO:teuthology.orchestra.run.smithi023.stdout:Repeating the final host check... 2024-03-28T15:42:18.792 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-28T15:42:18.792 INFO:teuthology.orchestra.run.smithi023.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-28T15:42:18.792 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl is present 2024-03-28T15:42:18.792 INFO:teuthology.orchestra.run.smithi023.stdout:lvcreate is present 2024-03-28T15:42:18.799 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-28T15:42:18.799 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-28T15:42:18.804 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-28T15:42:18.804 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:42:18.810 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout enabled 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout active 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Unit chronyd.service is enabled and running 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Host looks OK 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Cluster fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Acquiring lock 139756587442080 on /run/cephadm/a96af96e-ed19-11ee-95d0-87774f69a715.lock 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Lock 139756587442080 acquired on /run/cephadm/a96af96e-ed19-11ee-95d0-87774f69a715.lock 2024-03-28T15:42:18.817 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 3300 ... 2024-03-28T15:42:18.818 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying IP 172.21.15.23 port 6789 ... 2024-03-28T15:42:18.818 INFO:teuthology.orchestra.run.smithi023.stdout:Base mon IP(s) is [172.21.15.23:3300, 172.21.15.23:6789], mon addrv is [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-28T15:42:18.821 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.23 metric 100 2024-03-28T15:42:18.821 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.23 metric 100 2024-03-28T15:42:18.823 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-28T15:42:18.823 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-28T15:42:18.823 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T15:42:18.826 INFO:teuthology.orchestra.run.smithi023.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-28T15:42:18.827 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-28T15:42:18.827 INFO:teuthology.orchestra.run.smithi023.stdout:Mon IP `172.21.15.23` is in CIDR network `172.21.0.0/20` 2024-03-28T15:42:18.827 INFO:teuthology.orchestra.run.smithi023.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-28T15:42:18.827 INFO:teuthology.orchestra.run.smithi023.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-28T15:42:18.828 INFO:teuthology.orchestra.run.smithi023.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:42:19.273 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Getting image source signatures 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-28T15:42:19.274 INFO:teuthology.orchestra.run.smithi023.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-28T15:42:20.186 INFO:teuthology.orchestra.run.smithi023.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T15:42:20.187 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-28T15:42:20.187 INFO:teuthology.orchestra.run.smithi023.stdout:Extracting ceph user uid/gid from container image... 2024-03-28T15:42:21.119 INFO:teuthology.orchestra.run.smithi023.stdout:stat: stdout 167 167 2024-03-28T15:42:21.119 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial keys... 2024-03-28T15:42:22.106 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDdjwVmk+Z2GRAAryH5NeX04EHpDUVERnaOiQ== 2024-03-28T15:42:23.012 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDejwVmGRFgGxAA4/yciv4VwbMCCmG2Na0+0A== 2024-03-28T15:42:23.934 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph-authtool: stdout AQDfjwVmo5y2ExAAR5AVH5ihU7sVBcOiWIVATQ== 2024-03-28T15:42:23.935 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial monmap... 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:monmaptool for a [v2:172.21.15.23:3300,v1:172.21.15.23:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:setting min_mon_release = quincy 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: set fsid to a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:42:27.572 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mon... 2024-03-28T15:42:36.512 INFO:teuthology.orchestra.run.smithi023.stdout:create mon.a on 2024-03-28T15:42:37.024 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-28T15:42:37.278 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-28T15:42:37.460 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target. 2024-03-28T15:42:37.460 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target. 2024-03-28T15:42:37.689 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a 2024-03-28T15:42:37.689 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-28T15:42:37.843 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target.wants/ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715@.service. 2024-03-28T15:42:38.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:38 smithi023 systemd[1]: Started Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:42:38.305 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:38 smithi023 ceph-mon[32081]: mkfs a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:38.310 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:42:38.310 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T15:42:38.317 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:42:38.317 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:42:38.317 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T15:42:38.317 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T15:42:38.318 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon to start... 2024-03-28T15:42:38.318 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mon... 2024-03-28T15:42:38.561 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:38 smithi023 ceph-mon[32081]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: fsmap 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: mgrmap e1: no daemons active 2024-03-28T15:42:39.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:39 smithi023 ceph-mon[32081]: from='client.? 172.21.15.23:0/3816387283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout cluster: 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout id: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout services: 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.545227s) 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout data: 2024-03-28T15:42:39.478 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout pgs: 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f56946e2700 1 Processor -- start 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f56946e2700 1 -- start start 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f56946e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f56946e2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c108480 con 0x7f568c107a90 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f569247e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f569247e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55948/0 (socket says 172.21.15.23:55948) 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.765+0000 7f569247e700 1 -- 172.21.15.23:0/856103613 learned_addr learned my addr 172.21.15.23:0/856103613 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f569247e700 1 -- 172.21.15.23:0/856103613 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c108c90 con 0x7f568c107a90 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f569247e700 1 --2- 172.21.15.23:0/856103613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56880098a0 tx=0x7f5688009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b948dcbc58070f3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:39.479 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f569147c700 1 -- 172.21.15.23:0/856103613 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5688004160 con 0x7f568c107a90 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f569147c700 1 -- 172.21.15.23:0/856103613 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688004300 con 0x7f568c107a90 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f569147c700 1 -- 172.21.15.23:0/856103613 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56880036a0 con 0x7f568c107a90 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f56946e2700 1 -- 172.21.15.23:0/856103613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 msgr2=0x7f568c107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.766+0000 7f56946e2700 1 --2- 172.21.15.23:0/856103613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f56880098a0 tx=0x7f5688009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- 172.21.15.23:0/856103613 shutdown_connections 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 --2- 172.21.15.23:0/856103613 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c107eb0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- 172.21.15.23:0/856103613 >> 172.21.15.23:0/856103613 conn(0x7f568c1030d0 msgr2=0x7f568c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- 172.21.15.23:0/856103613 shutdown_connections 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- 172.21.15.23:0/856103613 wait complete. 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 Processor -- start 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- start start 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:39.480 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.767+0000 7f56946e2700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f568c1091c0 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f569247e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f569247e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55956/0 (socket says 172.21.15.23:55956) 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f569247e700 1 -- 172.21.15.23:0/3816387283 learned_addr learned my addr 172.21.15.23:0/3816387283 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f569247e700 1 -- 172.21.15.23:0/3816387283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f568c19b750 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f569247e700 1 --2- 172.21.15.23:0/3816387283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5688003db0 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5688011630 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5688011950 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f568c19b9e0 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f568c19e6d0 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f568801a460 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.768+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f568801b780 con 0x7f568c107a90 2024-03-28T15:42:39.481 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.769+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f568c0637c0 con 0x7f568c107a90 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.771+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5688031020 con 0x7f568c107a90 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.844+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f568c02dc80 con 0x7f568c107a90 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.844+0000 7f56837fe700 1 -- 172.21.15.23:0/3816387283 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f568801b9d0 con 0x7f568c107a90 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 msgr2=0x7f568c19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 --2- 172.21.15.23:0/3816387283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5688000c00 tx=0x7f5688004750 comp rx=0 tx=0).stop 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 shutdown_connections 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 --2- 172.21.15.23:0/3816387283 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f568c107a90 0x7f568c19b210 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 >> 172.21.15.23:0/3816387283 conn(0x7f568c1030d0 msgr2=0x7f568c191df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 shutdown_connections 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:38.845+0000 7f56946e2700 1 -- 172.21.15.23:0/3816387283 wait complete. 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:mon is available 2024-03-28T15:42:39.482 INFO:teuthology.orchestra.run.smithi023.stdout:Assimilating anything we can from ceph.conf... 2024-03-28T15:42:40.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:40 smithi023 ceph-mon[32081]: from='client.? 172.21.15.23:0/782472748' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T15:42:40.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:40 smithi023 ceph-mon[32081]: from='client.? 172.21.15.23:0/782472748' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.23:3300,v1:172.21.15.23:6789] 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T15:42:40.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.849+0000 7fbaadd06700 1 Processor -- start 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.849+0000 7fbaadd06700 1 -- start start 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaadd06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaadd06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa807be10 con 0x7fbaa8106360 2024-03-28T15:42:40.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55970/0 (socket says 172.21.15.23:55970) 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa77fe700 1 -- 172.21.15.23:0/2570375775 learned_addr learned my addr 172.21.15.23:0/2570375775 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa77fe700 1 -- 172.21.15.23:0/2570375775 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8106cc0 con 0x7fbaa8106360 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa77fe700 1 --2- 172.21.15.23:0/2570375775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fba980098a0 tx=0x7fba98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c0634ba86cac3c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.850+0000 7fbaa67fc700 1 -- 172.21.15.23:0/2570375775 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba98004160 con 0x7fbaa8106360 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaa67fc700 1 -- 172.21.15.23:0/2570375775 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98004300 con 0x7fbaa8106360 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaa67fc700 1 -- 172.21.15.23:0/2570375775 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba980036a0 con 0x7fbaa8106360 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 -- 172.21.15.23:0/2570375775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 msgr2=0x7fbaa8106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 --2- 172.21.15.23:0/2570375775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fba980098a0 tx=0x7fba98009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 -- 172.21.15.23:0/2570375775 shutdown_connections 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 --2- 172.21.15.23:0/2570375775 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa8106780 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 -- 172.21.15.23:0/2570375775 >> 172.21.15.23:0/2570375775 conn(0x7fbaa8101f20 msgr2=0x7fbaa8104360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 -- 172.21.15.23:0/2570375775 shutdown_connections 2024-03-28T15:42:40.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.851+0000 7fbaadd06700 1 -- 172.21.15.23:0/2570375775 wait complete. 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.852+0000 7fbaadd06700 1 Processor -- start 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.852+0000 7fbaadd06700 1 -- start start 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaadd06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaadd06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa81071f0 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa77fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55976/0 (socket says 172.21.15.23:55976) 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa77fe700 1 -- 172.21.15.23:0/782472748 learned_addr learned my addr 172.21.15.23:0/782472748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa77fe700 1 -- 172.21.15.23:0/782472748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa819d890 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa77fe700 1 --2- 172.21.15.23:0/782472748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba98003db0 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba98011630 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa819db20 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.853+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fba98011950 con 0x7fbaa8106360 2024-03-28T15:42:40.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.854+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa81a0810 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.854+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fba98011af0 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.854+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fba9801c450 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.855+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa80637c0 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.857+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fba98032020 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.931+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fbaa802dbb0 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.935+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7fba98004260 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.935+0000 7fbaa4ff9700 1 -- 172.21.15.23:0/782472748 <== mon.0 v2:172.21.15.23:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fba9801b450 con 0x7fbaa8106360 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 msgr2=0x7fbaa819d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 --2- 172.21.15.23:0/782472748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fba98000c00 tx=0x7fba98004750 comp rx=0 tx=0).stop 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 shutdown_connections 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 --2- 172.21.15.23:0/782472748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbaa8106360 0x7fbaa819d350 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:40.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 >> 172.21.15.23:0/782472748 conn(0x7fbaa8101f20 msgr2=0x7fbaa8193f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:40.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 shutdown_connections 2024-03-28T15:42:40.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:39.936+0000 7fbaadd06700 1 -- 172.21.15.23:0/782472748 wait complete. 2024-03-28T15:42:40.560 INFO:teuthology.orchestra.run.smithi023.stdout:Generating new minimal ceph.conf... 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b8b98700 1 Processor -- start 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b8b98700 1 -- start start 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b8b98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b8b98700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b407be30 con 0x7f95b4108580 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55982/0 (socket says 172.21.15.23:55982) 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b259c700 1 -- 172.21.15.23:0/1693830655 learned_addr learned my addr 172.21.15.23:0/1693830655 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.955+0000 7f95b259c700 1 -- 172.21.15.23:0/1693830655 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b410aeb0 con 0x7f95b4108580 2024-03-28T15:42:41.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b259c700 1 --2- 172.21.15.23:0/1693830655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f95a40098a0 tx=0x7f95a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec2f26cc6688dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b159a700 1 -- 172.21.15.23:0/1693830655 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95a4004160 con 0x7f95b4108580 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b159a700 1 -- 172.21.15.23:0/1693830655 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95a4004300 con 0x7f95b4108580 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b159a700 1 -- 172.21.15.23:0/1693830655 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95a4011b50 con 0x7f95b4108580 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 -- 172.21.15.23:0/1693830655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 msgr2=0x7f95b410a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 --2- 172.21.15.23:0/1693830655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f95a40098a0 tx=0x7f95a4009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 -- 172.21.15.23:0/1693830655 shutdown_connections 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 --2- 172.21.15.23:0/1693830655 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b410a970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 -- 172.21.15.23:0/1693830655 >> 172.21.15.23:0/1693830655 conn(0x7f95b4101f90 msgr2=0x7f95b41043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.956+0000 7f95b8b98700 1 -- 172.21.15.23:0/1693830655 shutdown_connections 2024-03-28T15:42:41.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.957+0000 7f95b8b98700 1 -- 172.21.15.23:0/1693830655 wait complete. 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.957+0000 7f95b8b98700 1 Processor -- start 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.957+0000 7f95b8b98700 1 -- start start 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.957+0000 7f95b8b98700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.957+0000 7f95b8b98700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b410be80 con 0x7f95b4108580 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55994/0 (socket says 172.21.15.23:55994) 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b259c700 1 -- 172.21.15.23:0/1099064260 learned_addr learned my addr 172.21.15.23:0/1099064260 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b259c700 1 -- 172.21.15.23:0/1099064260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b4108340 con 0x7f95b4108580 2024-03-28T15:42:41.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b259c700 1 --2- 172.21.15.23:0/1099064260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f95a4011b60 tx=0x7f95a4011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95a4003b20 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95a4003cc0 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b4106530 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95a4024840 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b4106a70 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f95a4022020 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.958+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f95a4011e00 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.959+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b40637c0 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:40.961+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f95a402d3f0 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.032+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f95b4106d20 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.032+0000 7f95a37fe700 1 -- 172.21.15.23:0/1099064260 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f95a4032030 con 0x7f95b4108580 2024-03-28T15:42:41.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 msgr2=0x7f95b4107e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 --2- 172.21.15.23:0/1099064260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f95a4011b60 tx=0x7f95a4011c40 comp rx=0 tx=0).stop 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 shutdown_connections 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 --2- 172.21.15.23:0/1099064260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95b4108580 0x7f95b4107e00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 >> 172.21.15.23:0/1099064260 conn(0x7f95b4101f90 msgr2=0x7f95b4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 shutdown_connections 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:41.033+0000 7f95b8b98700 1 -- 172.21.15.23:0/1099064260 wait complete. 2024-03-28T15:42:41.560 INFO:teuthology.orchestra.run.smithi023.stdout:Restarting the monitor... 2024-03-28T15:42:41.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:41 smithi023 ceph-mon[32081]: from='client.? 172.21.15.23:0/1099064260' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:42:41.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:41 smithi023 systemd[1]: Stopping Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:42:41.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:41 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a[32077]: 2024-03-28T15:42:41.810+0000 7fb1487c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T15:42:41.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:41 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a[32077]: 2024-03-28T15:42:41.810+0000 7fb1487c1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-28T15:42:41.862 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:41 smithi023 podman[32325]: 2024-03-28 15:42:41.861791976 +0000 UTC m=+0.145574901 container died dde9c7ac6697d7556a71690913bb3202d0c69f83580d34ce9598ee12f5c4e8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T15:42:42.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 podman[32325]: 2024-03-28 15:42:42.326893712 +0000 UTC m=+0.610676645 container cleanup dde9c7ac6697d7556a71690913bb3202d0c69f83580d34ce9598ee12f5c4e8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T15:42:42.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 bash[32325]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a 2024-03-28T15:42:42.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 podman[32340]: 2024-03-28 15:42:42.552629562 +0000 UTC m=+0.689141438 container remove dde9c7ac6697d7556a71690913bb3202d0c69f83580d34ce9598ee12f5c4e8c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T15:42:42.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T15:42:42.906 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 systemd[1]: Stopped Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:42:42.907 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 systemd[1]: Starting Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:42:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:42 smithi023 podman[32430]: 2024-03-28 15:42:42.806341278 +0000 UTC m=+0.018163319 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:43 smithi023 podman[32430]: 2024-03-28 15:42:43.861956695 +0000 UTC m=+1.073778735 container create 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 podman[32430]: 2024-03-28 15:42:44.056478651 +0000 UTC m=+1.268300696 container init 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 podman[32430]: 2024-03-28 15:42:44.060080071 +0000 UTC m=+1.271902111 container start 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: set uid:gid to 167:167 (ceph:ceph) 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: pidfile_write: ignore empty --pid-file 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: load: jerasure load: lrc 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: RocksDB version: 7.9.2 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Git sha 0 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-28T15:42:44.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: DB SUMMARY 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: DB Session ID: PUG7XGGTEEO6OZTBDN28 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: CURRENT file: CURRENT 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: IDENTITY file: IDENTITY 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100042 ; 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.error_if_exists: 0 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.create_if_missing: 0 2024-03-28T15:42:44.157 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.paranoid_checks: 1 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.env: 0x563d99ebcb20 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.fs: PosixFileSystem 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.info_log: 0x563d9c0e5920 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_file_opening_threads: 16 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.statistics: (nil) 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.use_fsync: 0 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_log_file_size: 0 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-28T15:42:44.159 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.keep_log_file_num: 1000 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.recycle_log_file_num: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_fallocate: 1 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_mmap_reads: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_mmap_writes: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.use_direct_reads: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.create_missing_column_families: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.db_log_dir: 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.wal_dir: 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-28T15:42:44.160 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.advise_random_on_open: 1 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.db_write_buffer_size: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.write_buffer_manager: 0x563d9c0b81e0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.rate_limiter: (nil) 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.wal_recovery_mode: 2 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enable_thread_tracking: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enable_pipelined_write: 0 2024-03-28T15:42:44.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.unordered_write: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.row_cache: None 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.wal_filter: None 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_ingest_behind: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.two_write_queues: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.manual_wal_flush: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.wal_compression: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.atomic_flush: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-28T15:42:44.162 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.log_readahead_size: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.best_efforts_recovery: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.allow_data_in_errors: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.db_host_id: __hostname__ 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_background_jobs: 2 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_background_compactions: -1 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_subcompactions: 1 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-28T15:42:44.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_total_wal_size: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_open_files: -1 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bytes_per_sync: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_readahead_size: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_background_flushes: -1 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Compression algorithms supported: 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kZSTD supported: 0 2024-03-28T15:42:44.164 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kXpressCompression supported: 0 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kLZ4HCCompression supported: 1 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kZlibCompression supported: 1 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kSnappyCompression supported: 1 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kLZ4Compression supported: 1 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: kBZip2Compression supported: 0 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.merge_operator: 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_filter: None 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_filter_factory: None 2024-03-28T15:42:44.165 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.sst_partitioner_factory: None 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563d9c0e5a20) 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks: 1 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: pin_top_level_index_and_filter: 1 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_type: 0 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_index_type: 0 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_shortening: 1 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: checksum: 4 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: no_block_cache: 0 2024-03-28T15:42:44.166 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache: 0x563d9c0eb090 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_name: BinnedLRUCache 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_options: 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: capacity : 536870912 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_shard_bits : 4 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: strict_capacity_limit : 0 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: high_pri_pool_ratio: 0.000 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_cache_compressed: (nil) 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: persistent_cache: (nil) 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size: 4096 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_size_deviation: 10 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_restart_interval: 16 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: index_block_restart_interval: 1 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: metadata_block_size: 4096 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: partition_filters: 0 2024-03-28T15:42:44.167 INFO:journalctl@ceph.mon.a.smithi023.stdout: use_delta_encoding: 1 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: filter_policy: bloomfilter 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: whole_key_filtering: 1 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: verify_compression: 0 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: read_amp_bytes_per_bit: 0 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: format_version: 5 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_index_compression: 1 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: block_align: 0 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: max_auto_readahead_size: 262144 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: prepopulate_block_cache: 0 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: initial_auto_readahead_size: 8192 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout: num_file_reads_for_auto_readahead: 2 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.write_buffer_size: 33554432 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_write_buffer_number: 2 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression: NoCompression 2024-03-28T15:42:44.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression: Disabled 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.prefix_extractor: nullptr 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.num_levels: 7 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:42:44.169 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.level: 32767 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.strategy: 0 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.enabled: false 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.target_file_size_base: 67108864 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-28T15:42:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.arena_block_size: 1048576 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-28T15:42:44.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.disable_auto_compactions: 0 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.table_properties_collectors: 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.inplace_update_support: 0 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-28T15:42:44.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.bloom_locality: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.max_successive_merges: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.paranoid_file_checks: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.force_consistency_checks: 1 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.report_bg_io_stats: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.ttl: 2592000 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enable_blob_files: false 2024-03-28T15:42:44.173 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.min_blob_size: 0 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_file_size: 268435456 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.blob_file_starting_level: 0 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 85c85266-6e67-4838-a2fe-62312523d065 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711640564091008, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-28T15:42:44.174 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711640564092168, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95260, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 316, "table_properties": {"data_size": 93259, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13771, "raw_average_key_size": 50, "raw_value_size": 85732, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 275, "num_filter_entries": 275, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711640564, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "85c85266-6e67-4838-a2fe-62312523d065", "db_session_id": "PUG7XGGTEEO6OZTBDN28", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711640564092236, "job": 1, "event": "recovery_finished"} 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563d9c1de000 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: DB pointer 0x563d9c1c8000 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** DB Stats ** 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-28T15:42:44.175 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: L0 2/0 94.86 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: Sum 2/0 94.86 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 111.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** Compaction Stats [default] ** 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 111.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-28T15:42:44.176 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Cumulative compaction: 0.00 GB write, 26.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Interval compaction: 0.00 GB write, 26.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache BinnedLRUCache@0x563d9c0eb090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: starting mon.a rank 0 at public addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] at bind addrs [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:44.177 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???) e1 preinit fsid a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).mds e1 new map 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).mds e1 print_map 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: e1 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: legacy client fscid: -1 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout: No filesystems configured 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 bash[32430]: 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 2024-03-28T15:42:44.178 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 systemd[1]: Started Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:42:44.179 INFO:teuthology.orchestra.run.smithi023.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-28T15:42:44.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-28T15:42:44.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: monmap e1: 1 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:42:44.408 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: fsmap 2024-03-28T15:42:44.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: osdmap e1: 0 total, 0 up, 0 in 2024-03-28T15:42:44.409 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:44 smithi023 ceph-mon[32448]: mgrmap e1: no daemons active 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.610+0000 7fd098b3f700 1 Processor -- start 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.610+0000 7fd098b3f700 1 -- start start 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.611+0000 7fd098b3f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.611+0000 7fd098b3f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09407c8d0 con 0x7fd09407dea0 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.611+0000 7fd09259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.611+0000 7fd09259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56010/0 (socket says 172.21.15.23:56010) 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.611+0000 7fd09259c700 1 -- 172.21.15.23:0/3126947415 learned_addr learned my addr 172.21.15.23:0/3126947415 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:45.236 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd09259c700 1 -- 172.21.15.23:0/3126947415 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09407ca50 con 0x7fd09407dea0 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd09259c700 1 --2- 172.21.15.23:0/3126947415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0840098a0 tx=0x7fd084009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14218c4afc7f93e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd09159a700 1 -- 172.21.15.23:0/3126947415 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd084004160 con 0x7fd09407dea0 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd09159a700 1 -- 172.21.15.23:0/3126947415 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd084004300 con 0x7fd09407dea0 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd09159a700 1 -- 172.21.15.23:0/3126947415 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd084011b50 con 0x7fd09407dea0 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.612+0000 7fd098b3f700 1 -- 172.21.15.23:0/3126947415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 msgr2=0x7fd09407c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 --2- 172.21.15.23:0/3126947415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd0840098a0 tx=0x7fd084009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 -- 172.21.15.23:0/3126947415 shutdown_connections 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 --2- 172.21.15.23:0/3126947415 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09407c300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 -- 172.21.15.23:0/3126947415 >> 172.21.15.23:0/3126947415 conn(0x7fd0941030d0 msgr2=0x7fd0941054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 -- 172.21.15.23:0/3126947415 shutdown_connections 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 -- 172.21.15.23:0/3126947415 wait complete. 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 Processor -- start 2024-03-28T15:42:45.237 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.613+0000 7fd098b3f700 1 -- start start 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd098b3f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd098b3f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09410e080 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd09259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd09259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56018/0 (socket says 172.21.15.23:56018) 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd09259c700 1 -- 172.21.15.23:0/1784913837 learned_addr learned my addr 172.21.15.23:0/1784913837 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd09259c700 1 -- 172.21.15.23:0/1784913837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09419e1d0 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd09259c700 1 --2- 172.21.15.23:0/1784913837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd084009fb0 tx=0x7fd084011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0840038b0 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09419e460 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd084011e50 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd084019550 con 0x7fd09407dea0 2024-03-28T15:42:45.238 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.614+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0941a1150 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.615+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd084019be0 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.615+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd084024730 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.615+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0940637c0 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.618+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fd084024930 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.687+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd0941a17e0 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.689+0000 7fd0837fe700 1 -- 172.21.15.23:0/1784913837 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd084003a50 con 0x7fd09407dea0 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 msgr2=0x7fd09419dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 --2- 172.21.15.23:0/1784913837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd084009fb0 tx=0x7fd084011bf0 comp rx=0 tx=0).stop 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 shutdown_connections 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 --2- 172.21.15.23:0/1784913837 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd09407dea0 0x7fd09419dc90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:45.239 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 >> 172.21.15.23:0/1784913837 conn(0x7fd0941030d0 msgr2=0x7fd094103c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:45.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 shutdown_connections 2024-03-28T15:42:45.240 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:44.690+0000 7fd098b3f700 1 -- 172.21.15.23:0/1784913837 wait complete. 2024-03-28T15:42:45.240 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-28T15:42:45.261 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:42:45.262 INFO:teuthology.orchestra.run.smithi023.stdout:Creating mgr... 2024-03-28T15:42:45.262 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-28T15:42:45.262 INFO:teuthology.orchestra.run.smithi023.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-28T15:42:45.498 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x 2024-03-28T15:42:45.498 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-28T15:42:45.646 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715.target.wants/ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-a96af96e-ed19-11ee-95d0-87774f69a715@.service. 2024-03-28T15:42:45.973 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1784913837' entity='client.admin' 2024-03-28T15:42:46.179 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:42:46.179 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T15:42:46.186 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:42:46.187 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:42:46.187 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T15:42:46.187 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to enable service . firewalld.service is not available 2024-03-28T15:42:46.194 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:42:46.195 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T15:42:46.201 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:42:46.202 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:42:46.202 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T15:42:46.202 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-28T15:42:46.202 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr to start... 2024-03-28T15:42:46.202 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr... 2024-03-28T15:42:47.156 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:46 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3074923758' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:42:47.156 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:46.759+0000 7f8032824200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:42:47.553 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:42:47.554 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:42:47.555 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:42:47.556 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:42:38.303087+0000", 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:42:47.557 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.619+0000 7fcff5923700 1 Processor -- start 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcff5923700 1 -- start start 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcff5923700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcff5923700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff0108480 con 0x7fcff0107a90 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56056/0 (socket says 172.21.15.23:56056) 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcfeeffd700 1 -- 172.21.15.23:0/2701125927 learned_addr learned my addr 172.21.15.23:0/2701125927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcfeeffd700 1 -- 172.21.15.23:0/2701125927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff0108c90 con 0x7fcff0107a90 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.620+0000 7fcfeeffd700 1 --2- 172.21.15.23:0/2701125927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00098a0 tx=0x7fcfe0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc96d957b4f88a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.621+0000 7fcfedffb700 1 -- 172.21.15.23:0/2701125927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe0004160 con 0x7fcff0107a90 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.621+0000 7fcfedffb700 1 -- 172.21.15.23:0/2701125927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe0004300 con 0x7fcff0107a90 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.621+0000 7fcfedffb700 1 -- 172.21.15.23:0/2701125927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe0011b50 con 0x7fcff0107a90 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.621+0000 7fcff5923700 1 -- 172.21.15.23:0/2701125927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 msgr2=0x7fcff0107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:47.558 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.621+0000 7fcff5923700 1 --2- 172.21.15.23:0/2701125927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00098a0 tx=0x7fcfe0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 -- 172.21.15.23:0/2701125927 shutdown_connections 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 --2- 172.21.15.23:0/2701125927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff0107eb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 -- 172.21.15.23:0/2701125927 >> 172.21.15.23:0/2701125927 conn(0x7fcff01030d0 msgr2=0x7fcff01054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 -- 172.21.15.23:0/2701125927 shutdown_connections 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 -- 172.21.15.23:0/2701125927 wait complete. 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 Processor -- start 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.622+0000 7fcff5923700 1 -- start start 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff5923700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff5923700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcff010c040 con 0x7fcff0107a90 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcfeeffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56064/0 (socket says 172.21.15.23:56064) 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcfeeffd700 1 -- 172.21.15.23:0/3074923758 learned_addr learned my addr 172.21.15.23:0/3074923758 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcfeeffd700 1 -- 172.21.15.23:0/3074923758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcff019bff0 con 0x7fcff0107a90 2024-03-28T15:42:47.559 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcfeeffd700 1 --2- 172.21.15.23:0/3074923758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009fb0 tx=0x7fcfe0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe00038b0 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcff019c280 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcfe0011e50 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcfe0019550 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.623+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcff019ef70 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.624+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fcfe0019be0 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.624+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcfe000bb10 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.625+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcff00637c0 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.627+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fcfe0032070 con 0x7fcff0107a90 2024-03-28T15:42:47.560 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.701+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcff019f140 con 0x7fcff0107a90 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.701+0000 7fcff4921700 1 -- 172.21.15.23:0/3074923758 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fcfe000bd10 con 0x7fcff0107a90 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 msgr2=0x7fcff019bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 --2- 172.21.15.23:0/3074923758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0009fb0 tx=0x7fcfe0011bf0 comp rx=0 tx=0).stop 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 shutdown_connections 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 --2- 172.21.15.23:0/3074923758 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcff0107a90 0x7fcff019bab0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 >> 172.21.15.23:0/3074923758 conn(0x7fcff01030d0 msgr2=0x7fcff0103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 shutdown_connections 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:46.702+0000 7fcff5923700 1 -- 172.21.15.23:0/3074923758 wait complete. 2024-03-28T15:42:47.561 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (1/15)... 2024-03-28T15:42:47.765 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:47.428+0000 7f8032824200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:42:47.765 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:47.522+0000 7f8032824200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:42:47.765 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:47.668+0000 7f8032824200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:42:47.765 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:47.765+0000 7f8032824200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:42:48.117 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:47.849+0000 7f8032824200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:42:48.117 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:48.117+0000 7f8032824200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:42:48.413 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:48.229+0000 7f8032824200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:42:48.413 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:48.316+0000 7f8032824200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:42:48.413 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:48.413+0000 7f8032824200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:42:48.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:48.590+0000 7f8032824200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:42:49.884 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:49.590+0000 7f8032824200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:42:49.884 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:49.724+0000 7f8032824200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:42:50.168 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1076926820' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:42:50.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:50.066+0000 7f8032824200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:42:50.168 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:50.167+0000 7f8032824200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:42:50.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:50.373+0000 7f8032824200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:42:50.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:42:50.706 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:42:50.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:42:50.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:42:38.303087+0000", 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:42:50.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab6d236700 1 Processor -- start 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab6d236700 1 -- start start 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab6d236700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab6d236700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68107360 con 0x7fab68106970 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab66d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab66d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55632/0 (socket says 172.21.15.23:55632) 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.987+0000 7fab66d9d700 1 -- 172.21.15.23:0/3119702351 learned_addr learned my addr 172.21.15.23:0/3119702351 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.988+0000 7fab66d9d700 1 -- 172.21.15.23:0/3119702351 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68107b70 con 0x7fab68106970 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.988+0000 7fab66d9d700 1 --2- 172.21.15.23:0/3119702351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab5000bb40 tx=0x7fab5000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72f2aa9567a7eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.988+0000 7fab65d9b700 1 -- 172.21.15.23:0/3119702351 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab5000dbe0 con 0x7fab68106970 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.988+0000 7fab65d9b700 1 -- 172.21.15.23:0/3119702351 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab500044f0 con 0x7fab68106970 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.988+0000 7fab65d9b700 1 -- 172.21.15.23:0/3119702351 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab50010470 con 0x7fab68106970 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 -- 172.21.15.23:0/3119702351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 msgr2=0x7fab68106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:50.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 --2- 172.21.15.23:0/3119702351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab5000bb40 tx=0x7fab5000be50 comp rx=0 tx=0).stop 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 -- 172.21.15.23:0/3119702351 shutdown_connections 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 --2- 172.21.15.23:0/3119702351 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab68106d90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 -- 172.21.15.23:0/3119702351 >> 172.21.15.23:0/3119702351 conn(0x7fab68101f70 msgr2=0x7fab68104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 -- 172.21.15.23:0/3119702351 shutdown_connections 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.989+0000 7fab6d236700 1 -- 172.21.15.23:0/3119702351 wait complete. 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab6d236700 1 Processor -- start 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab6d236700 1 -- start start 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab6d236700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab6d236700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6807afb0 con 0x7fab68106970 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab66d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab66d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55642/0 (socket says 172.21.15.23:55642) 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.990+0000 7fab66d9d700 1 -- 172.21.15.23:0/1076926820 learned_addr learned my addr 172.21.15.23:0/1076926820 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:50.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab66d9d700 1 -- 172.21.15.23:0/1076926820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6819e160 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab66d9d700 1 --2- 172.21.15.23:0/1076926820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab5000b340 tx=0x7fab500040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab50010720 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fab5001d070 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab50021c20 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6819e3f0 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab681a10e0 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fab5001f030 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.991+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fab500193f0 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.992+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab680637c0 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:49.994+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fab50015070 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.069+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fab68192d70 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.070+0000 7fab5ffff700 1 -- 172.21.15.23:0/1076926820 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fab50021410 con 0x7fab68106970 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 msgr2=0x7fab6819dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:50.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 --2- 172.21.15.23:0/1076926820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fab5000b340 tx=0x7fab500040b0 comp rx=0 tx=0).stop 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 shutdown_connections 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 --2- 172.21.15.23:0/1076926820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fab68106970 0x7fab6819dc20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 >> 172.21.15.23:0/1076926820 conn(0x7fab68101f70 msgr2=0x7fab6818fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 shutdown_connections 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:50.071+0000 7fab6d236700 1 -- 172.21.15.23:0/1076926820 wait complete. 2024-03-28T15:42:50.713 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (2/15)... 2024-03-28T15:42:50.855 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:50.564+0000 7f8032824200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:42:51.115 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:50.855+0000 7f8032824200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:42:51.115 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:51.115+0000 7f8032824200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:42:51.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:51.356+0000 7f8032824200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:42:52.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:52.180+0000 7f8032824200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:42:53.032 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:52.764+0000 7f8032824200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:42:53.032 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:52.864+0000 7f8032824200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:53.089+0000 7f8032824200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: Activating manager daemon x 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: mgrmap e2: x(active, starting, since 0.0055447s) 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:42:53.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: Manager daemon x is now available 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' 2024-03-28T15:42:53.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:53 smithi023 ceph-mon[32448]: from='mgr.14100 172.21.15.23:0/3768201524' entity='mgr.x' 2024-03-28T15:42:53.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:42:53.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:42:53.718 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:42:53.727 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:42:53.728 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:42:38.303087+0000", 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.132+0000 7efcb4c64700 1 Processor -- start 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.132+0000 7efcb4c64700 1 -- start start 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcb4c64700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:53.729 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcb4c64700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0107360 con 0x7efcb0106970 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55688/0 (socket says 172.21.15.23:55688) 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcae59c700 1 -- 172.21.15.23:0/3261863927 learned_addr learned my addr 172.21.15.23:0/3261863927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcae59c700 1 -- 172.21.15.23:0/3261863927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0107b70 con 0x7efcb0106970 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.133+0000 7efcae59c700 1 --2- 172.21.15.23:0/3261863927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efca00098a0 tx=0x7efca0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba39c70f65ad2bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcad59a700 1 -- 172.21.15.23:0/3261863927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efca0004160 con 0x7efcb0106970 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcad59a700 1 -- 172.21.15.23:0/3261863927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efca0004300 con 0x7efcb0106970 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcad59a700 1 -- 172.21.15.23:0/3261863927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efca0011b50 con 0x7efcb0106970 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcb4c64700 1 -- 172.21.15.23:0/3261863927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 msgr2=0x7efcb0106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcb4c64700 1 --2- 172.21.15.23:0/3261863927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efca00098a0 tx=0x7efca0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcb4c64700 1 -- 172.21.15.23:0/3261863927 shutdown_connections 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.134+0000 7efcb4c64700 1 --2- 172.21.15.23:0/3261863927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0106d90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:53.730 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 -- 172.21.15.23:0/3261863927 >> 172.21.15.23:0/3261863927 conn(0x7efcb0101f70 msgr2=0x7efcb0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 -- 172.21.15.23:0/3261863927 shutdown_connections 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 -- 172.21.15.23:0/3261863927 wait complete. 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 Processor -- start 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 -- start start 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.135+0000 7efcb4c64700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb007afb0 con 0x7efcb0106970 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efcae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efcae59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55716/0 (socket says 172.21.15.23:55716) 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efcae59c700 1 -- 172.21.15.23:0/393228974 learned_addr learned my addr 172.21.15.23:0/393228974 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efcae59c700 1 -- 172.21.15.23:0/393228974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb0199d50 con 0x7efcb0106970 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efcae59c700 1 --2- 172.21.15.23:0/393228974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efca0000c00 tx=0x7efca0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.136+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efca0003b20 con 0x7efcb0106970 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.137+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efca0003cc0 con 0x7efcb0106970 2024-03-28T15:42:53.731 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.137+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb0199fe0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.137+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efca0024910 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.137+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb019ccd0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.138+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 2) v1 ==== 45862+0+0 (secure 0 0 0) 0x7efca0022020 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.138+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb00637c0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.138+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efca0011e00 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.141+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7efca002cdf0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.215+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efcb019cea0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.215+0000 7efc9f7fe700 1 -- 172.21.15.23:0/393228974 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7efcb019cea0 con 0x7efcb0106970 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 msgr2=0x7efcb0199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 --2- 172.21.15.23:0/393228974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efca0000c00 tx=0x7efca0011c10 comp rx=0 tx=0).stop 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 shutdown_connections 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 --2- 172.21.15.23:0/393228974 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcb0106970 0x7efcb0199810 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 >> 172.21.15.23:0/393228974 conn(0x7efcb0101f70 msgr2=0x7efcb018fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:53.732 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 shutdown_connections 2024-03-28T15:42:53.733 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:53.216+0000 7efcb4c64700 1 -- 172.21.15.23:0/393228974 wait complete. 2024-03-28T15:42:53.733 INFO:teuthology.orchestra.run.smithi023.stdout:mgr not available, waiting (3/15)... 2024-03-28T15:42:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/393228974' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:42:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:54 smithi023 ceph-mon[32448]: mgrmap e3: x(active, since 1.00872s) 2024-03-28T15:42:56.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:56 smithi023 ceph-mon[32448]: mgrmap e4: x(active, since 2s) 2024-03-28T15:42:57.013 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:57.013 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "health": { 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 0 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "a" 2024-03-28T15:42:57.014 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-28T15:42:57.015 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-28T15:42:57.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-28T15:42:57.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-28T15:42:57.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-28T15:42:57.024 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-28T15:42:57.025 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "restful" 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ], 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-28T15:42:57.026 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "modified": "2024-03-28T15:42:38.303087+0000", 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout }, 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb568eeb700 1 Processor -- start 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb568eeb700 1 -- start start 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb568eeb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb568eeb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb564108480 con 0x7fb564107a90 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb56259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb56259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55742/0 (socket says 172.21.15.23:55742) 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.152+0000 7fb56259c700 1 -- 172.21.15.23:0/1277591503 learned_addr learned my addr 172.21.15.23:0/1277591503 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.153+0000 7fb56259c700 1 -- 172.21.15.23:0/1277591503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564108c90 con 0x7fb564107a90 2024-03-28T15:42:57.027 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.153+0000 7fb56259c700 1 --2- 172.21.15.23:0/1277591503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5540098a0 tx=0x7fb554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd5dfc6f955bbbef server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:57.028 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.153+0000 7fb56159a700 1 -- 172.21.15.23:0/1277591503 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb554004160 con 0x7fb564107a90 2024-03-28T15:42:57.028 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.153+0000 7fb56159a700 1 -- 172.21.15.23:0/1277591503 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb554004300 con 0x7fb564107a90 2024-03-28T15:42:57.028 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.153+0000 7fb56159a700 1 -- 172.21.15.23:0/1277591503 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb554011b50 con 0x7fb564107a90 2024-03-28T15:42:57.028 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 -- 172.21.15.23:0/1277591503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 msgr2=0x7fb564107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 --2- 172.21.15.23:0/1277591503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5540098a0 tx=0x7fb554009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 -- 172.21.15.23:0/1277591503 shutdown_connections 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 --2- 172.21.15.23:0/1277591503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb564107eb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 -- 172.21.15.23:0/1277591503 >> 172.21.15.23:0/1277591503 conn(0x7fb5641030d0 msgr2=0x7fb5641054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 -- 172.21.15.23:0/1277591503 shutdown_connections 2024-03-28T15:42:57.029 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.154+0000 7fb568eeb700 1 -- 172.21.15.23:0/1277591503 wait complete. 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb568eeb700 1 Processor -- start 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb568eeb700 1 -- start start 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb568eeb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb568eeb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56410c040 con 0x7fb564107a90 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb56259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb56259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55758/0 (socket says 172.21.15.23:55758) 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb56259c700 1 -- 172.21.15.23:0/3442550609 learned_addr learned my addr 172.21.15.23:0/3442550609 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb56259c700 1 -- 172.21.15.23:0/3442550609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56407e090 con 0x7fb564107a90 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb56259c700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb554011850 tx=0x7fb554011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.155+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb554003af0 con 0x7fb564107a90 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.156+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb554020070 con 0x7fb564107a90 2024-03-28T15:42:57.030 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.156+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56407c280 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.156+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5540195a0 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.156+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56407c7c0 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.156+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fb5540198a0 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.157+0000 7fb5537fe700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 0x7fb54c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.157+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb554011e00 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.157+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5640637c0 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.157+0000 7fb561d9b700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 0x7fb54c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.157+0000 7fb561d9b700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 0x7fb54c03b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb558006fd0 tx=0x7fb558006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.162+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb554003c90 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.451+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb56407ca40 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.452+0000 7fb5537fe700 1 -- 172.21.15.23:0/3442550609 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb554023530 con 0x7fb564107a90 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 msgr2=0x7fb54c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 0x7fb54c03b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb558006fd0 tx=0x7fb558006e40 comp rx=0 tx=0).stop 2024-03-28T15:42:57.031 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 msgr2=0x7fb56407db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb554011850 tx=0x7fb554011c10 comp rx=0 tx=0).stop 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 shutdown_connections 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fb54c038ed0 0x7fb54c03b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 --2- 172.21.15.23:0/3442550609 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb564107a90 0x7fb56407db50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 >> 172.21.15.23:0/3442550609 conn(0x7fb5641030d0 msgr2=0x7fb564103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 shutdown_connections 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:56.454+0000 7fb568eeb700 1 -- 172.21.15.23:0/3442550609 wait complete. 2024-03-28T15:42:57.032 INFO:teuthology.orchestra.run.smithi023.stdout:mgr is available 2024-03-28T15:42:57.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:57 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3442550609' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [global] 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout fsid = a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-28T15:42:58.182 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout [osd] 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.387+0000 7fe354f76700 1 Processor -- start 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.387+0000 7fe354f76700 1 -- start start 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.387+0000 7fe354f76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.387+0000 7fe354f76700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe350108480 con 0x7fe350107a90 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55772/0 (socket says 172.21.15.23:55772) 2024-03-28T15:42:58.183 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34e59c700 1 -- 172.21.15.23:0/2417233150 learned_addr learned my addr 172.21.15.23:0/2417233150 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34e59c700 1 -- 172.21.15.23:0/2417233150 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe350108c90 con 0x7fe350107a90 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34e59c700 1 --2- 172.21.15.23:0/2417233150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe3380098a0 tx=0x7fe338009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bc0cb102db9cb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34d59a700 1 -- 172.21.15.23:0/2417233150 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe338004160 con 0x7fe350107a90 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34d59a700 1 -- 172.21.15.23:0/2417233150 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe338004300 con 0x7fe350107a90 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.388+0000 7fe34d59a700 1 -- 172.21.15.23:0/2417233150 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe338011b50 con 0x7fe350107a90 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 -- 172.21.15.23:0/2417233150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 msgr2=0x7fe350107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 --2- 172.21.15.23:0/2417233150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe3380098a0 tx=0x7fe338009bb0 comp rx=0 tx=0).stop 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 -- 172.21.15.23:0/2417233150 shutdown_connections 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 --2- 172.21.15.23:0/2417233150 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe350107eb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 -- 172.21.15.23:0/2417233150 >> 172.21.15.23:0/2417233150 conn(0x7fe3501030d0 msgr2=0x7fe3501054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 -- 172.21.15.23:0/2417233150 shutdown_connections 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.389+0000 7fe354f76700 1 -- 172.21.15.23:0/2417233150 wait complete. 2024-03-28T15:42:58.184 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 Processor -- start 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 -- start start 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35010c040 con 0x7fe350107a90 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe34e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55774/0 (socket says 172.21.15.23:55774) 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe34e59c700 1 -- 172.21.15.23:0/3848688522 learned_addr learned my addr 172.21.15.23:0/3848688522 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe34e59c700 1 -- 172.21.15.23:0/3848688522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35007e090 con 0x7fe350107a90 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe34e59c700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe338009fb0 tx=0x7fe338011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3380038b0 con 0x7fe350107a90 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35007c280 con 0x7fe350107a90 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe338011e50 con 0x7fe350107a90 2024-03-28T15:42:58.185 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe338019550 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.390+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35007c7c0 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.391+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fe338019850 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.391+0000 7fe3477fe700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 0x7fe33c03b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.391+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe33804d1c0 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.391+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3500637c0 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.392+0000 7fe34dd9b700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 0x7fe33c03b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.392+0000 7fe34dd9b700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 0x7fe33c03b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe340006fd0 tx=0x7fe340006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.396+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe33804cc20 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.601+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fe35007ca40 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.602+0000 7fe3477fe700 1 -- 172.21.15.23:0/3848688522 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fe338032020 con 0x7fe350107a90 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 msgr2=0x7fe33c03b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 0x7fe33c03b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe340006fd0 tx=0x7fe340006e40 comp rx=0 tx=0).stop 2024-03-28T15:42:58.186 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 msgr2=0x7fe35007db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe338009fb0 tx=0x7fe338011bf0 comp rx=0 tx=0).stop 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 shutdown_connections 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fe33c038e80 0x7fe33c03b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 --2- 172.21.15.23:0/3848688522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe350107a90 0x7fe35007db50 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 >> 172.21.15.23:0/3848688522 conn(0x7fe3501030d0 msgr2=0x7fe350103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 shutdown_connections 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:57.604+0000 7fe354f76700 1 -- 172.21.15.23:0/3848688522 wait complete. 2024-03-28T15:42:58.187 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling cephadm module... 2024-03-28T15:42:58.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3848688522' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-28T15:42:59.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:42:59 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/315120607' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-28T15:42:59.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setuser ceph since I am not root 2024-03-28T15:42:59.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setgroup ceph since I am not root 2024-03-28T15:42:59.782 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.593+0000 7f9cdf697700 1 Processor -- start 2024-03-28T15:42:59.782 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.593+0000 7f9cdf697700 1 -- start start 2024-03-28T15:42:59.782 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.593+0000 7f9cdf697700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.593+0000 7f9cdf697700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd807be10 con 0x7f9cd8106350 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cdd433700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cdd433700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55776/0 (socket says 172.21.15.23:55776) 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cdd433700 1 -- 172.21.15.23:0/1911287380 learned_addr learned my addr 172.21.15.23:0/1911287380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cdd433700 1 -- 172.21.15.23:0/1911287380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd8106cb0 con 0x7f9cd8106350 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cdd433700 1 --2- 172.21.15.23:0/1911287380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009e80 tx=0x7f9cd000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfba2d71b152d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cd7fff700 1 -- 172.21.15.23:0/1911287380 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cd0017070 con 0x7f9cd8106350 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cd7fff700 1 -- 172.21.15.23:0/1911287380 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd000b8c0 con 0x7f9cd8106350 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.594+0000 7f9cd7fff700 1 -- 172.21.15.23:0/1911287380 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cd00045c0 con 0x7f9cd8106350 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 -- 172.21.15.23:0/1911287380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 msgr2=0x7f9cd8106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 --2- 172.21.15.23:0/1911287380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9cd0009e80 tx=0x7f9cd000b330 comp rx=0 tx=0).stop 2024-03-28T15:42:59.783 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 -- 172.21.15.23:0/1911287380 shutdown_connections 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 --2- 172.21.15.23:0/1911287380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd8106770 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 -- 172.21.15.23:0/1911287380 >> 172.21.15.23:0/1911287380 conn(0x7f9cd8101f70 msgr2=0x7f9cd8104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 -- 172.21.15.23:0/1911287380 shutdown_connections 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.595+0000 7f9cdf697700 1 -- 172.21.15.23:0/1911287380 wait complete. 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdf697700 1 Processor -- start 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdf697700 1 -- start start 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdf697700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdf697700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd8107d40 con 0x7f9cd8106350 2024-03-28T15:42:59.784 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdd433700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdd433700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55784/0 (socket says 172.21.15.23:55784) 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdd433700 1 -- 172.21.15.23:0/315120607 learned_addr learned my addr 172.21.15.23:0/315120607 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdd433700 1 -- 172.21.15.23:0/315120607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd819e070 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdd433700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00072a0 tx=0x7f9cd000bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cd0020020 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cd0003b50 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd819e300 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.596+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cd0017040 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.597+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd81a0ff0 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.597+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f9cd0023020 con 0x7f9cd8106350 2024-03-28T15:42:59.785 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.597+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd80637c0 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.597+0000 7f9cd67fc700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 0x7f9cc003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.598+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cd000f050 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.598+0000 7f9cdcc32700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 0x7f9cc003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.598+0000 7f9cdcc32700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 0x7f9cc003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc006fd0 tx=0x7f9ccc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.603+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9cd002a020 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:58.867+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9cd802dd70 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.174+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f9cd002d350 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.176+0000 7f9cd67fc700 1 -- 172.21.15.23:0/315120607 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9cd0012980 con 0x7f9cd8106350 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 msgr2=0x7f9cc003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:59.786 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 0x7f9cc003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ccc006fd0 tx=0x7f9ccc006e40 comp rx=0 tx=0).stop 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 msgr2=0x7f9cd819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9cd00072a0 tx=0x7f9cd000bd50 comp rx=0 tx=0).stop 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 shutdown_connections 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f9cc0038ed0 0x7f9cc003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 --2- 172.21.15.23:0/315120607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9cd8106350 0x7f9cd819db30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 >> 172.21.15.23:0/315120607 conn(0x7f9cd8101f70 msgr2=0x7f9cd8194810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 shutdown_connections 2024-03-28T15:42:59.787 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:42:59.177+0000 7f9cdf697700 1 -- 172.21.15.23:0/315120607 wait complete. 2024-03-28T15:43:00.074 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:59.744+0000 7f967bea0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:43:00.074 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:42:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:42:59.842+0000 7f967bea0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:43:00.387 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/315120607' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-28T15:43:00.387 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-mon[32448]: mgrmap e5: x(active, since 6s) 2024-03-28T15:43:00.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:00.537+0000 7f967bea0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:43:00.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:00.632+0000 7f967bea0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:43:01.063 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:00.776+0000 7f967bea0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:43:01.063 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:00.873+0000 7f967bea0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:43:01.063 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:00.958+0000 7f967bea0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.173+0000 7f33c1611700 1 Processor -- start 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.173+0000 7f33c1611700 1 -- start start 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.173+0000 7f33c1611700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.173+0000 7f33c1611700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc108470 con 0x7f33bc107a80 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.174+0000 7f33baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.174+0000 7f33baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55806/0 (socket says 172.21.15.23:55806) 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.174+0000 7f33baffd700 1 -- 172.21.15.23:0/4162398782 learned_addr learned my addr 172.21.15.23:0/4162398782 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:01.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.174+0000 7f33baffd700 1 -- 172.21.15.23:0/4162398782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc108c80 con 0x7f33bc107a80 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.174+0000 7f33baffd700 1 --2- 172.21.15.23:0/4162398782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0098a0 tx=0x7f33ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23af87bda333dd2e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.181+0000 7f33b9ffb700 1 -- 172.21.15.23:0/4162398782 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33ac004160 con 0x7f33bc107a80 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.181+0000 7f33b9ffb700 1 -- 172.21.15.23:0/4162398782 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33ac004300 con 0x7f33bc107a80 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.181+0000 7f33b9ffb700 1 -- 172.21.15.23:0/4162398782 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33ac011b50 con 0x7f33bc107a80 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 -- 172.21.15.23:0/4162398782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 msgr2=0x7f33bc107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 --2- 172.21.15.23:0/4162398782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0098a0 tx=0x7f33ac009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 -- 172.21.15.23:0/4162398782 shutdown_connections 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 --2- 172.21.15.23:0/4162398782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc107ea0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 -- 172.21.15.23:0/4162398782 >> 172.21.15.23:0/4162398782 conn(0x7f33bc103020 msgr2=0x7f33bc105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 -- 172.21.15.23:0/4162398782 shutdown_connections 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.182+0000 7f33c1611700 1 -- 172.21.15.23:0/4162398782 wait complete. 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.183+0000 7f33c1611700 1 Processor -- start 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.183+0000 7f33c1611700 1 -- start start 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.183+0000 7f33c1611700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:01.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.183+0000 7f33c1611700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33bc10c030 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33baffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55820/0 (socket says 172.21.15.23:55820) 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33baffd700 1 -- 172.21.15.23:0/1786234169 learned_addr learned my addr 172.21.15.23:0/1786234169 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33baffd700 1 -- 172.21.15.23:0/1786234169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc19bf70 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33baffd700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f33ac011670 tx=0x7f33ac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33ac0039c0 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33ac011e50 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33bc19c200 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f33ac01a5c0 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.184+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33bc19eef0 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.185+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f33ac01a870 con 0x7f33bc107a80 2024-03-28T15:43:01.074 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.185+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33bc0637c0 con 0x7f33bc107a80 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.185+0000 7f33a3fff700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 0x7f33a403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.186+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f33ac04d0e0 con 0x7f33bc107a80 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.186+0000 7f33ba7fc700 1 -- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 msgr2=0x7f33a403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.186+0000 7f33ba7fc700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 0x7f33a403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.190+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33ac020240 con 0x7f33bc107a80 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.387+0000 7f33ba7fc700 1 -- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 msgr2=0x7f33a403b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.387+0000 7f33ba7fc700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 0x7f33a403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.483+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f33bc19f0c0 con 0x7f33bc107a80 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.484+0000 7f33a3fff700 1 -- 172.21.15.23:0/1786234169 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f33bc19f0c0 con 0x7f33bc107a80 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 msgr2=0x7f33a403b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 0x7f33a403b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 msgr2=0x7f33bc19ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:01.075 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f33ac011670 tx=0x7f33ac011b60 comp rx=0 tx=0).stop 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 shutdown_connections 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7f33a4038f20 0x7f33a403b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 --2- 172.21.15.23:0/1786234169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f33bc107a80 0x7f33bc19ba30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 >> 172.21.15.23:0/1786234169 conn(0x7f33bc103020 msgr2=0x7f33bc103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 shutdown_connections 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:00.486+0000 7f33c1611700 1 -- 172.21.15.23:0/1786234169 wait complete. 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-28T15:43:01.076 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 5... 2024-03-28T15:43:01.348 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1786234169' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T15:43:01.349 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:01.232+0000 7f967bea0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:43:01.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:01.348+0000 7f967bea0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:43:01.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:01.437+0000 7f967bea0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:43:01.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:01.537+0000 7f967bea0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:43:02.097 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:01.714+0000 7f967bea0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:43:03.165 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:02.695+0000 7f967bea0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:43:03.165 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:02.828+0000 7f967bea0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:43:03.165 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:03.165+0000 7f967bea0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:43:03.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:03.256+0000 7f967bea0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:43:03.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:03.457+0000 7f967bea0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:43:03.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:03.646+0000 7f967bea0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:43:04.200 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:03.939+0000 7f967bea0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:43:04.500 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:04.200+0000 7f967bea0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:43:04.500 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:04.442+0000 7f967bea0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:43:05.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:05.274+0000 7f967bea0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:43:06.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:05.854+0000 7f967bea0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:43:06.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:05.955+0000 7f967bea0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:06.178+0000 7f967bea0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: Active manager daemon x restarted 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: Activating manager daemon x 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: osdmap e2: 0 total, 0 up, 0 in 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: mgrmap e6: x(active, starting, since 0.0124597s) 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:43:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:43:06.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:43:06.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: Manager daemon x is now available 2024-03-28T15:43:06.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:06 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: Found migration_current of "None". Setting to last migration. 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: mgrmap e7: x(active, since 1.0143s) 2024-03-28T15:43:07.509 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:07 smithi023 ceph-mon[32448]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T15:43:07.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.490+0000 7fa1b78b7700 1 Processor -- start 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.490+0000 7fa1b78b7700 1 -- start start 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b78b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b78b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0108470 con 0x7fa1b0107a80 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b5653700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b5653700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55832/0 (socket says 172.21.15.23:55832) 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b5653700 1 -- 172.21.15.23:0/185330061 learned_addr learned my addr 172.21.15.23:0/185330061 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b5653700 1 -- 172.21.15.23:0/185330061 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0108c80 con 0x7fa1b0107a80 2024-03-28T15:43:07.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1b5653700 1 --2- 172.21.15.23:0/185330061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0098a0 tx=0x7fa1ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756f48a7645a779f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.491+0000 7fa1a7fff700 1 -- 172.21.15.23:0/185330061 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ac004160 con 0x7fa1b0107a80 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1a7fff700 1 -- 172.21.15.23:0/185330061 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1ac004300 con 0x7fa1b0107a80 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1a7fff700 1 -- 172.21.15.23:0/185330061 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ac011b50 con 0x7fa1b0107a80 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 -- 172.21.15.23:0/185330061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 msgr2=0x7fa1b0107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/185330061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0098a0 tx=0x7fa1ac009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 -- 172.21.15.23:0/185330061 shutdown_connections 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/185330061 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b0107ea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 -- 172.21.15.23:0/185330061 >> 172.21.15.23:0/185330061 conn(0x7fa1b0103080 msgr2=0x7fa1b01054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 -- 172.21.15.23:0/185330061 shutdown_connections 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.492+0000 7fa1b78b7700 1 -- 172.21.15.23:0/185330061 wait complete. 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b78b7700 1 Processor -- start 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b78b7700 1 -- start start 2024-03-28T15:43:07.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b78b7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b78b7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b010c030 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b5653700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b5653700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55840/0 (socket says 172.21.15.23:55840) 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b5653700 1 -- 172.21.15.23:0/2028432784 learned_addr learned my addr 172.21.15.23:0/2028432784 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.493+0000 7fa1b5653700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b019bf50 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1b5653700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac011670 tx=0x7fa1ac011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ac0039c0 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa1ac011e50 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b019c1e0 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1ac01a5c0 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.494+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b019eed0 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa1ac01a870 con 0x7fa1b0107a80 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1a67fc700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa19c03baf0 con 0x7fa19c038f20 2024-03-28T15:43:07.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1b4e52700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa1ac04d3c0 con 0x7fa1b0107a80 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.495+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.696+0000 7fa1b4e52700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:01.696+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:02.097+0000 7fa1b4e52700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:02.097+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:02.898+0000 7fa1b4e52700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:02.898+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:04.499+0000 7fa1b4e52700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/3484701507 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:04.499+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:06.202+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fa1ac01abb0 con 0x7fa1b0107a80 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:06.202+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 msgr2=0x7fa19c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:43:07.957 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:06.202+0000 7fa1a67fc700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.203+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fa1ac04efc0 con 0x7fa1b0107a80 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.203+0000 7fa1a67fc700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 0x7fa19c03ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.203+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1ac016e70 con 0x7fa19c03c810 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.203+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 0x7fa19c03ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.204+0000 7fa1b4e52700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 0x7fa19c03ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0003a10 tx=0x7fa1a00092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.205+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fa1ac016e70 con 0x7fa19c03c810 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.211+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa1b00637c0 con 0x7fa19c03c810 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.211+0000 7fa1a67fc700 1 -- 172.21.15.23:0/2028432784 <== mgr.14120 v2:172.21.15.23:6800/4208679988 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fa1b00637c0 con 0x7fa19c03c810 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 msgr2=0x7fa19c03ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 0x7fa19c03ec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1a0003a10 tx=0x7fa1a00092b0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 msgr2=0x7fa1b019ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac011670 tx=0x7fa1ac011b60 comp rx=0 tx=0).stop 2024-03-28T15:43:07.958 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 shutdown_connections 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa19c03c810 0x7fa19c03ec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1b0107a80 0x7fa1b019ba10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 --2- 172.21.15.23:0/2028432784 >> [v2:172.21.15.23:6800/3484701507,v1:172.21.15.23:6801/3484701507] conn(0x7fa19c038f20 0x7fa19c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 >> 172.21.15.23:0/2028432784 conn(0x7fa1b0103080 msgr2=0x7fa1b0103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 shutdown_connections 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:07.212+0000 7fa1b78b7700 1 -- 172.21.15.23:0/2028432784 wait complete. 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 5 is available 2024-03-28T15:43:07.959 INFO:teuthology.orchestra.run.smithi023.stdout:Setting orchestrator backend to cephadm... 2024-03-28T15:43:08.624 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:08 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:07] ENGINE Bus STARTING 2024-03-28T15:43:08.624 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:08 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:07] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T15:43:08.624 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:08 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:07] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T15:43:08.624 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:08 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:07] ENGINE Bus STARTED 2024-03-28T15:43:08.624 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:08 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.391+0000 7f31314a1700 1 Processor -- start 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.391+0000 7f31314a1700 1 -- start start 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.391+0000 7f31314a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f31314a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c07c890 con 0x7f312c07de60 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f312affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f312affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55902/0 (socket says 172.21.15.23:55902) 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f312affd700 1 -- 172.21.15.23:0/3605455748 learned_addr learned my addr 172.21.15.23:0/3605455748 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:09.191 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f312affd700 1 -- 172.21.15.23:0/3605455748 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c07ca10 con 0x7f312c07de60 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f312affd700 1 --2- 172.21.15.23:0/3605455748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e80 tx=0x7f311c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=353f4815c62ed247 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f3129ffb700 1 -- 172.21.15.23:0/3605455748 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f311c017070 con 0x7f312c07de60 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f3129ffb700 1 -- 172.21.15.23:0/3605455748 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f311c00b8c0 con 0x7f312c07de60 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.392+0000 7f3129ffb700 1 -- 172.21.15.23:0/3605455748 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f311c0045c0 con 0x7f312c07de60 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 -- 172.21.15.23:0/3605455748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 msgr2=0x7f312c07c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 --2- 172.21.15.23:0/3605455748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f311c009e80 tx=0x7f311c00b330 comp rx=0 tx=0).stop 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 -- 172.21.15.23:0/3605455748 shutdown_connections 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 --2- 172.21.15.23:0/3605455748 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c07c2c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 -- 172.21.15.23:0/3605455748 >> 172.21.15.23:0/3605455748 conn(0x7f312c103020 msgr2=0x7f312c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 -- 172.21.15.23:0/3605455748 shutdown_connections 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.393+0000 7f31314a1700 1 -- 172.21.15.23:0/3605455748 wait complete. 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f31314a1700 1 Processor -- start 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f31314a1700 1 -- start start 2024-03-28T15:43:09.192 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f31314a1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f31314a1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312c10e0b0 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f312affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f312affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55910/0 (socket says 172.21.15.23:55910) 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f312affd700 1 -- 172.21.15.23:0/1178968560 learned_addr learned my addr 172.21.15.23:0/1178968560 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.394+0000 7f312affd700 1 -- 172.21.15.23:0/1178968560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c19e0c0 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f312affd700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f311c004350 tx=0x7f311c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f311c020020 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f311c003c10 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312c19e350 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f311c017040 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.395+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312c1a1040 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.396+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f311c023020 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.396+0000 7f3123fff700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 0x7f311803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.396+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312c0637c0 con 0x7f312c07de60 2024-03-28T15:43:09.193 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.396+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f311c00f050 con 0x7f312c07de60 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.396+0000 7f312a7fc700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 0x7f311803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.397+0000 7f312a7fc700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 0x7f311803b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3114006fd0 tx=0x7f3114006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.401+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f311c02a020 con 0x7f312c07de60 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.624+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f312c1041d0 con 0x7f3118038f70 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.635+0000 7f3123fff700 1 -- 172.21.15.23:0/1178968560 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f312c1041d0 con 0x7f3118038f70 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 msgr2=0x7f311803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 0x7f311803b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3114006fd0 tx=0x7f3114006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 msgr2=0x7f312c19db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f311c004350 tx=0x7f311c00bd50 comp rx=0 tx=0).stop 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 shutdown_connections 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f3118038f70 0x7f311803b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 --2- 172.21.15.23:0/1178968560 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f312c07de60 0x7f312c19db80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 >> 172.21.15.23:0/1178968560 conn(0x7f312c103020 msgr2=0x7f312c103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:09.194 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 shutdown_connections 2024-03-28T15:43:09.195 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:08.638+0000 7f31314a1700 1 -- 172.21.15.23:0/1178968560 wait complete. 2024-03-28T15:43:09.527 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:09 smithi023 ceph-mon[32448]: mgrmap e8: x(active, since 2s) 2024-03-28T15:43:09.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:09 smithi023 ceph-mon[32448]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:09.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:09 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:09.528 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:09 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:10.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:10 smithi023 ceph-mon[32448]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:10.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-28T15:43:10.431 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1aa8f700 1 Processor -- start 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1aa8f700 1 -- start start 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1aa8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1aa8f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e14107340 con 0x7f1e14106950 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1882b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1882b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54418/0 (socket says 172.21.15.23:54418) 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1882b700 1 -- 172.21.15.23:0/1265116092 learned_addr learned my addr 172.21.15.23:0/1265116092 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.632+0000 7f1e1882b700 1 -- 172.21.15.23:0/1265116092 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e14107b50 con 0x7f1e14106950 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1882b700 1 --2- 172.21.15.23:0/1265116092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e080098a0 tx=0x7f1e08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e930ad955716b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e137fe700 1 -- 172.21.15.23:0/1265116092 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e08004160 con 0x7f1e14106950 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e137fe700 1 -- 172.21.15.23:0/1265116092 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e08004300 con 0x7f1e14106950 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e137fe700 1 -- 172.21.15.23:0/1265116092 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e08011b50 con 0x7f1e14106950 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/1265116092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 msgr2=0x7f1e14106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:10.432 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/1265116092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1e080098a0 tx=0x7f1e08009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/1265116092 shutdown_connections 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/1265116092 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e14106d70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/1265116092 >> 172.21.15.23:0/1265116092 conn(0x7f1e14101f10 msgr2=0x7f1e14104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/1265116092 shutdown_connections 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.633+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/1265116092 wait complete. 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1aa8f700 1 Processor -- start 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1aa8f700 1 -- start start 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1aa8f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1aa8f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1410af00 con 0x7f1e14106950 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1882b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1882b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54426/0 (socket says 172.21.15.23:54426) 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.634+0000 7f1e1882b700 1 -- 172.21.15.23:0/4290935951 learned_addr learned my addr 172.21.15.23:0/4290935951 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e1882b700 1 -- 172.21.15.23:0/4290935951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1419bf70 con 0x7f1e14106950 2024-03-28T15:43:10.433 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e1882b700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e08011670 tx=0x7f1e08011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e080039c0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e08011e50 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1419c200 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e0801a5c0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.635+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1419eea0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.636+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1e0801a8c0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.636+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e140637c0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.636+0000 7f1e11ffb700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 0x7f1e0403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.636+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1e0804d3d0 con 0x7f1e14106950 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.636+0000 7f1e13fff700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 0x7f1e0403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.637+0000 7f1e13fff700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 0x7f1e0403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e00006fd0 tx=0x7f1e00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:10.434 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.641+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e08019c70 con 0x7f1e14106950 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.847+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e140008d0 con 0x7f1e04038ed0 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.847+0000 7f1e11ffb700 1 -- 172.21.15.23:0/4290935951 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f1e140008d0 con 0x7f1e04038ed0 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.849+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 msgr2=0x7f1e0403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.849+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 0x7f1e0403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e00006fd0 tx=0x7f1e00006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.849+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 msgr2=0x7f1e1419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.849+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1e08011670 tx=0x7f1e08011b60 comp rx=0 tx=0).stop 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 shutdown_connections 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1e04038ed0 0x7f1e0403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 --2- 172.21.15.23:0/4290935951 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1e14106950 0x7f1e1419ba30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 >> 172.21.15.23:0/4290935951 conn(0x7f1e14101f10 msgr2=0x7f1e14102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 shutdown_connections 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:09.850+0000 7f1e1aa8f700 1 -- 172.21.15.23:0/4290935951 wait complete. 2024-03-28T15:43:10.435 INFO:teuthology.orchestra.run.smithi023.stdout:Generating ssh key... 2024-03-28T15:43:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-mon[32448]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:11.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-mon[32448]: Generating ssh key... 2024-03-28T15:43:11.970 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: Generating public/private rsa key pair. 2024-03-28T15:43:11.970 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: Your identification has been saved in /tmp/tmpnajeqzr6/key. 2024-03-28T15:43:11.970 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: Your public key has been saved in /tmp/tmpnajeqzr6/key.pub. 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: The key fingerprint is: 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: SHA256:5QPi4p23SWuGbV3x2zYPO/N/Ot0asT7gR7B6t6GXHjU ceph-a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: The key's randomart image is: 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: +---[RSA 3072]----+ 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | | 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | | 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | . . . | 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | . . + o | 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | . . S o =.E.| 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | . o . .+ o+.| 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | . ooo. + o*=.| 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | .o=+o oo@B*| 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: | ++ ..=B@O| 2024-03-28T15:43:11.971 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: +----[SHA256]-----+ 2024-03-28T15:43:12.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.829+0000 7fb1bf2d3700 1 Processor -- start 2024-03-28T15:43:12.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.829+0000 7fb1bf2d3700 1 -- start start 2024-03-28T15:43:12.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.829+0000 7fb1bf2d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:12.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.829+0000 7fb1bf2d3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b807be90 con 0x7fb1b8108520 2024-03-28T15:43:12.262 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1bd06f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1bd06f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54428/0 (socket says 172.21.15.23:54428) 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1bd06f700 1 -- 172.21.15.23:0/103628088 learned_addr learned my addr 172.21.15.23:0/103628088 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1bd06f700 1 -- 172.21.15.23:0/103628088 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b810ae50 con 0x7fb1b8108520 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1bd06f700 1 --2- 172.21.15.23:0/103628088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009e80 tx=0x7fb1a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43b1e5d59a46c2f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1b7fff700 1 -- 172.21.15.23:0/103628088 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8017070 con 0x7fb1b8108520 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1b7fff700 1 -- 172.21.15.23:0/103628088 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1a800b8c0 con 0x7fb1b8108520 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.830+0000 7fb1b7fff700 1 -- 172.21.15.23:0/103628088 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1a80045c0 con 0x7fb1b8108520 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/103628088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 msgr2=0x7fb1b810a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/103628088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8009e80 tx=0x7fb1a800b330 comp rx=0 tx=0).stop 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/103628088 shutdown_connections 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/103628088 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b810a910 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/103628088 >> 172.21.15.23:0/103628088 conn(0x7fb1b8101f90 msgr2=0x7fb1b81043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:12.263 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/103628088 shutdown_connections 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/103628088 wait complete. 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 Processor -- start 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.831+0000 7fb1bf2d3700 1 -- start start 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bf2d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bf2d3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b810be80 con 0x7fb1b8108520 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bd06f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bd06f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54438/0 (socket says 172.21.15.23:54438) 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bd06f700 1 -- 172.21.15.23:0/2669796227 learned_addr learned my addr 172.21.15.23:0/2669796227 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bd06f700 1 -- 172.21.15.23:0/2669796227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b81082e0 con 0x7fb1b8108520 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bd06f700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004350 tx=0x7fb1a800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8020020 con 0x7fb1b8108520 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1a8003c10 con 0x7fb1b8108520 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb1a8017040 con 0x7fb1b8108520 2024-03-28T15:43:12.264 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b81064d0 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.832+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b8106a10 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.833+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb1a8023020 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.833+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b80637c0 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.833+0000 7fb1b67fc700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 0x7fb1a003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.833+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb1a800f050 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.834+0000 7fb1bc86e700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 0x7fb1a003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.834+0000 7fb1bc86e700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 0x7fb1a003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:10.838+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb1a802a020 con 0x7fb1b8108520 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.042+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1b8103090 con 0x7fb1a0038f70 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.711+0000 7fb1b67fc700 1 -- 172.21.15.23:0/2669796227 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1b8103090 con 0x7fb1a0038f70 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 msgr2=0x7fb1a003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 0x7fb1a003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac006fd0 tx=0x7fb1ac006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 msgr2=0x7fb1b8107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:12.265 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004350 tx=0x7fb1a800bd50 comp rx=0 tx=0).stop 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 shutdown_connections 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb1a0038f70 0x7fb1a003b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 --2- 172.21.15.23:0/2669796227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1b8108520 0x7fb1b8107da0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 >> 172.21.15.23:0/2669796227 conn(0x7fb1b8101f90 msgr2=0x7fb1b8102980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 shutdown_connections 2024-03-28T15:43:12.266 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:11.713+0000 7fb1bf2d3700 1 -- 172.21.15.23:0/2669796227 wait complete. 2024-03-28T15:43:12.892 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:12 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:12.892 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:12 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:13.444 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.676+0000 7f8d5a755700 1 Processor -- start 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.676+0000 7f8d5a755700 1 -- start start 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.676+0000 7f8d5a755700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.676+0000 7f8d5a755700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d54108480 con 0x7f8d54107a90 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.676+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54452/0 (socket says 172.21.15.23:54452) 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d53fff700 1 -- 172.21.15.23:0/1812654074 learned_addr learned my addr 172.21.15.23:0/1812654074 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d53fff700 1 -- 172.21.15.23:0/1812654074 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54108c90 con 0x7f8d54107a90 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d53fff700 1 --2- 172.21.15.23:0/1812654074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d440098a0 tx=0x7f8d44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a6d8b4d122ba3f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d52ffd700 1 -- 172.21.15.23:0/1812654074 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d44004160 con 0x7f8d54107a90 2024-03-28T15:43:13.445 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d52ffd700 1 -- 172.21.15.23:0/1812654074 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d44004300 con 0x7f8d54107a90 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.677+0000 7f8d52ffd700 1 -- 172.21.15.23:0/1812654074 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d44011b50 con 0x7f8d54107a90 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- 172.21.15.23:0/1812654074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 msgr2=0x7f8d54107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 --2- 172.21.15.23:0/1812654074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8d440098a0 tx=0x7f8d44009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- 172.21.15.23:0/1812654074 shutdown_connections 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 --2- 172.21.15.23:0/1812654074 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d54107eb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- 172.21.15.23:0/1812654074 >> 172.21.15.23:0/1812654074 conn(0x7f8d541030d0 msgr2=0x7f8d541054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- 172.21.15.23:0/1812654074 shutdown_connections 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- 172.21.15.23:0/1812654074 wait complete. 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 Processor -- start 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.678+0000 7f8d5a755700 1 -- start start 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d5a755700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d5a755700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5410c040 con 0x7f8d54107a90 2024-03-28T15:43:13.446 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d53fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54468/0 (socket says 172.21.15.23:54468) 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d53fff700 1 -- 172.21.15.23:0/131471924 learned_addr learned my addr 172.21.15.23:0/131471924 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d53fff700 1 -- 172.21.15.23:0/131471924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5407e090 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d53fff700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009fb0 tx=0x7f8d44011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.679+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d440038b0 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.680+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5407c280 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.680+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d44011e50 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.680+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d44019550 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.680+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5407c7c0 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.680+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f8d44019850 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.681+0000 7f8d517fa700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 0x7f8d3c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.681+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8d4404d3c0 con 0x7f8d54107a90 2024-03-28T15:43:13.447 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.681+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d540637c0 con 0x7f8d54107a90 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.681+0000 7f8d537fe700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 0x7f8d3c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.681+0000 7f8d537fe700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 0x7f8d3c03b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.686+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d4400f2e0 con 0x7f8d54107a90 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.892+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5407d0f0 con 0x7f8d3c038ed0 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.893+0000 7f8d517fa700 1 -- 172.21.15.23:0/131471924 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f8d5407d0f0 con 0x7f8d3c038ed0 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 msgr2=0x7f8d3c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 0x7f8d3c03b390 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8d48006fd0 tx=0x7f8d48006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 msgr2=0x7f8d5407db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009fb0 tx=0x7f8d44011bf0 comp rx=0 tx=0).stop 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 shutdown_connections 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f8d3c038ed0 0x7f8d3c03b390 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 --2- 172.21.15.23:0/131471924 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d54107a90 0x7f8d5407db50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 >> 172.21.15.23:0/131471924 conn(0x7f8d541030d0 msgr2=0x7f8d54103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:13.448 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 shutdown_connections 2024-03-28T15:43:13.449 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:12.895+0000 7f8d5a755700 1 -- 172.21.15.23:0/131471924 wait complete. 2024-03-28T15:43:13.449 INFO:teuthology.orchestra.run.smithi023.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-28T15:43:13.449 INFO:teuthology.orchestra.run.smithi023.stdout:Adding key to root@localhost authorized_keys... 2024-03-28T15:43:13.449 INFO:teuthology.orchestra.run.smithi023.stdout:Adding host smithi023... 2024-03-28T15:43:13.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:13 smithi023 ceph-mon[32448]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:14.914 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:14 smithi023 ceph-mon[32448]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:16.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:15 smithi023 ceph-mon[32448]: Deploying cephadm binary to smithi023 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Added host 'smithi023' with addr '172.21.15.23' 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3c6765700 1 Processor -- start 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3c6765700 1 -- start start 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3c6765700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3c6765700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01073a0 con 0x7fb3c01069b0 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3bffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3bffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54474/0 (socket says 172.21.15.23:54474) 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.847+0000 7fb3bffff700 1 -- 172.21.15.23:0/2444324625 learned_addr learned my addr 172.21.15.23:0/2444324625 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3bffff700 1 -- 172.21.15.23:0/2444324625 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c0107bb0 con 0x7fb3c01069b0 2024-03-28T15:43:19.069 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3bffff700 1 --2- 172.21.15.23:0/2444324625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00098a0 tx=0x7fb3b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e587d10ed942a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3beffd700 1 -- 172.21.15.23:0/2444324625 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0004160 con 0x7fb3c01069b0 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3beffd700 1 -- 172.21.15.23:0/2444324625 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3b0004300 con 0x7fb3c01069b0 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3beffd700 1 -- 172.21.15.23:0/2444324625 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0011b50 con 0x7fb3c01069b0 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3c6765700 1 -- 172.21.15.23:0/2444324625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 msgr2=0x7fb3c0106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.848+0000 7fb3c6765700 1 --2- 172.21.15.23:0/2444324625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00098a0 tx=0x7fb3b0009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 -- 172.21.15.23:0/2444324625 shutdown_connections 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 --2- 172.21.15.23:0/2444324625 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0106dd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 -- 172.21.15.23:0/2444324625 >> 172.21.15.23:0/2444324625 conn(0x7fb3c0101f90 msgr2=0x7fb3c01043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 -- 172.21.15.23:0/2444324625 shutdown_connections 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 -- 172.21.15.23:0/2444324625 wait complete. 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 Processor -- start 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.849+0000 7fb3c6765700 1 -- start start 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3c6765700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:19.070 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3c6765700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c010af60 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bffff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54484/0 (socket says 172.21.15.23:54484) 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bffff700 1 -- 172.21.15.23:0/3883247314 learned_addr learned my addr 172.21.15.23:0/3883247314 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bffff700 1 -- 172.21.15.23:0/3883247314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c01920a0 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bffff700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0003b20 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3b0003cc0 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c0190290 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0024910 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.850+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c01907d0 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.851+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb3b0022020 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.851+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c00637c0 con 0x7fb3c01069b0 2024-03-28T15:43:19.071 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.851+0000 7fb3bd7fa700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 0x7fb3a803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.851+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb3b0011e00 con 0x7fb3c01069b0 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.852+0000 7fb3bf7fe700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 0x7fb3a803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.852+0000 7fb3bf7fe700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 0x7fb3a803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4006fd0 tx=0x7fb3b4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:13.856+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3b0016260 con 0x7fb3c01069b0 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:14.067+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi023", "addr": "172.21.15.23", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3c0195970 con 0x7fb3a8038f70 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.480+0000 7fb3bd7fa700 1 -- 172.21.15.23:0/3883247314 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb3c0195970 con 0x7fb3a8038f70 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 msgr2=0x7fb3a803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 0x7fb3a803b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4006fd0 tx=0x7fb3b4006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 msgr2=0x7fb3c0191b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b0011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 shutdown_connections 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fb3a8038f70 0x7fb3a803b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:19.072 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 --2- 172.21.15.23:0/3883247314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3c01069b0 0x7fb3c0191b60 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:19.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 >> 172.21.15.23:0/3883247314 conn(0x7fb3c0101f90 msgr2=0x7fb3c0102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:19.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 shutdown_connections 2024-03-28T15:43:19.073 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:18.483+0000 7fb3c6765700 1 -- 172.21.15.23:0/3883247314 wait complete. 2024-03-28T15:43:19.073 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mon service... 2024-03-28T15:43:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:19 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:19 smithi023 ceph-mon[32448]: Added host smithi023 2024-03-28T15:43:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:19 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:20.426 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.483+0000 7fa5370c9700 1 Processor -- start 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa5370c9700 1 -- start start 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa5370c9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa5370c9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53007be30 con 0x7fa530108540 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa534e65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa534e65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33914/0 (socket says 172.21.15.23:33914) 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa534e65700 1 -- 172.21.15.23:0/2144786914 learned_addr learned my addr 172.21.15.23:0/2144786914 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa534e65700 1 -- 172.21.15.23:0/2144786914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53010ae70 con 0x7fa530108540 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.484+0000 7fa534e65700 1 --2- 172.21.15.23:0/2144786914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5240098a0 tx=0x7fa524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603cf05a009d32fa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa52f7fe700 1 -- 172.21.15.23:0/2144786914 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa524004160 con 0x7fa530108540 2024-03-28T15:43:20.427 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa52f7fe700 1 -- 172.21.15.23:0/2144786914 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa524004300 con 0x7fa530108540 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa52f7fe700 1 -- 172.21.15.23:0/2144786914 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa524011b50 con 0x7fa530108540 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 -- 172.21.15.23:0/2144786914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 msgr2=0x7fa53010a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 --2- 172.21.15.23:0/2144786914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa5240098a0 tx=0x7fa524009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 -- 172.21.15.23:0/2144786914 shutdown_connections 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 --2- 172.21.15.23:0/2144786914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa53010a930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 -- 172.21.15.23:0/2144786914 >> 172.21.15.23:0/2144786914 conn(0x7fa530101f70 msgr2=0x7fa530104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 -- 172.21.15.23:0/2144786914 shutdown_connections 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.485+0000 7fa5370c9700 1 -- 172.21.15.23:0/2144786914 wait complete. 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa5370c9700 1 Processor -- start 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa5370c9700 1 -- start start 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa5370c9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa5370c9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa53010be40 con 0x7fa530108540 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa534e65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:20.428 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa534e65700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33918/0 (socket says 172.21.15.23:33918) 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa534e65700 1 -- 172.21.15.23:0/1182856428 learned_addr learned my addr 172.21.15.23:0/1182856428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.486+0000 7fa534e65700 1 -- 172.21.15.23:0/1182856428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530108300 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa534e65700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa524011670 tx=0x7fa524011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5240039c0 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa524011e50 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5301064f0 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa52401a5c0 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.487+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa530106a30 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.488+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fa524024430 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.488+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5300637c0 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.488+0000 7fa52dffb700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 0x7fa52003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.488+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa52404d320 con 0x7fa530108540 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.488+0000 7fa52ffff700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 0x7fa52003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:20.429 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.489+0000 7fa52ffff700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 0x7fa52003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006fd0 tx=0x7fa51c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.494+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa524017ca0 con 0x7fa530108540 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.714+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa530103190 con 0x7fa520038ed0 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.719+0000 7fa52dffb700 1 -- 172.21.15.23:0/1182856428 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa530103190 con 0x7fa520038ed0 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 msgr2=0x7fa52003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 0x7fa52003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006fd0 tx=0x7fa51c006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 msgr2=0x7fa530107dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa524011670 tx=0x7fa524011b60 comp rx=0 tx=0).stop 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 shutdown_connections 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fa520038ed0 0x7fa52003b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 --2- 172.21.15.23:0/1182856428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa530108540 0x7fa530107dc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 >> 172.21.15.23:0/1182856428 conn(0x7fa530101f70 msgr2=0x7fa530102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 shutdown_connections 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:19.721+0000 7fa5370c9700 1 -- 172.21.15.23:0/1182856428 wait complete. 2024-03-28T15:43:20.430 INFO:teuthology.orchestra.run.smithi023.stdout:Deploying unmanaged mgr service... 2024-03-28T15:43:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:20 smithi023 ceph-mon[32448]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:20 smithi023 ceph-mon[32448]: Saving service mon spec with placement count:5 2024-03-28T15:43:21.023 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:20 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:22 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:22 smithi023 ceph-mon[32448]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:22 smithi023 ceph-mon[32448]: Saving service mgr spec with placement count:2 2024-03-28T15:43:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:22 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:22 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:22.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-28T15:43:22.680 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.815+0000 7f26b0069700 1 Processor -- start 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26b0069700 1 -- start start 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26b0069700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26b0069700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a8108470 con 0x7f26a8107a80 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26ade05700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26ade05700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33920/0 (socket says 172.21.15.23:33920) 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26ade05700 1 -- 172.21.15.23:0/3251520660 learned_addr learned my addr 172.21.15.23:0/3251520660 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.816+0000 7f26ade05700 1 -- 172.21.15.23:0/3251520660 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a8108c80 con 0x7f26a8107a80 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26ade05700 1 --2- 172.21.15.23:0/3251520660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26a40098a0 tx=0x7f26a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54e28eda78cc870 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26ace03700 1 -- 172.21.15.23:0/3251520660 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a4004160 con 0x7f26a8107a80 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26ace03700 1 -- 172.21.15.23:0/3251520660 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a4004300 con 0x7f26a8107a80 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26ace03700 1 -- 172.21.15.23:0/3251520660 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a4011b20 con 0x7f26a8107a80 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26b0069700 1 -- 172.21.15.23:0/3251520660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 msgr2=0x7f26a8107ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:22.681 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26b0069700 1 --2- 172.21.15.23:0/3251520660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26a40098a0 tx=0x7f26a4009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26b0069700 1 -- 172.21.15.23:0/3251520660 shutdown_connections 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26b0069700 1 --2- 172.21.15.23:0/3251520660 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a8107ea0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.817+0000 7f26b0069700 1 -- 172.21.15.23:0/3251520660 >> 172.21.15.23:0/3251520660 conn(0x7f26a8103020 msgr2=0x7f26a8105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 -- 172.21.15.23:0/3251520660 shutdown_connections 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 -- 172.21.15.23:0/3251520660 wait complete. 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 Processor -- start 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 -- start start 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.818+0000 7f26b0069700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a810c030 con 0x7f26a8107a80 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26ade05700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26ade05700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33922/0 (socket says 172.21.15.23:33922) 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26ade05700 1 -- 172.21.15.23:0/788242189 learned_addr learned my addr 172.21.15.23:0/788242189 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26ade05700 1 -- 172.21.15.23:0/788242189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a807e090 con 0x7f26a8107a80 2024-03-28T15:43:22.682 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26ade05700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26a40117e0 tx=0x7f26a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a4003bf0 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a401a910 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a807c280 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26a40249a0 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.819+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a807c740 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.820+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f26a4022020 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.820+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a80637c0 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.820+0000 7f269effd700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 0x7f269403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.820+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f26a404d260 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.821+0000 7f26ad604700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 0x7f269403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.821+0000 7f26ad604700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 0x7f269403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2698006fd0 tx=0x7f2698006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:21.825+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26a4011e20 con 0x7f26a8107a80 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.046+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f26a807cf20 con 0x7f2694038f20 2024-03-28T15:43:22.683 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.051+0000 7f269effd700 1 -- 172.21.15.23:0/788242189 <== mgr.14120 v2:172.21.15.23:6800/4208679988 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f26a807cf20 con 0x7f2694038f20 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 msgr2=0x7f269403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 0x7f269403b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2698006fd0 tx=0x7f2698006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 msgr2=0x7f26a807db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26a40117e0 tx=0x7f26a4011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 shutdown_connections 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.053+0000 7f26b0069700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f2694038f20 0x7f269403b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.054+0000 7f26b0069700 1 --2- 172.21.15.23:0/788242189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26a8107a80 0x7f26a807db50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.054+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 >> 172.21.15.23:0/788242189 conn(0x7f26a8103020 msgr2=0x7f26a8103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.054+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 shutdown_connections 2024-03-28T15:43:22.684 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:22.054+0000 7f26b0069700 1 -- 172.21.15.23:0/788242189 wait complete. 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.128+0000 7f1b31093700 1 Processor -- start 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.128+0000 7f1b31093700 1 -- start start 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.128+0000 7f1b31093700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.128+0000 7f1b31093700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c108480 con 0x7f1b2c107a90 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.128+0000 7f1b2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33930/0 (socket says 172.21.15.23:33930) 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b2ad9d700 1 -- 172.21.15.23:0/1749271251 learned_addr learned my addr 172.21.15.23:0/1749271251 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b2ad9d700 1 -- 172.21.15.23:0/1749271251 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c108c90 con 0x7f1b2c107a90 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b2ad9d700 1 --2- 172.21.15.23:0/1749271251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0098a0 tx=0x7f1b1c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13157ae9821b5939 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b29d9b700 1 -- 172.21.15.23:0/1749271251 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b1c004160 con 0x7f1b2c107a90 2024-03-28T15:43:24.383 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b29d9b700 1 -- 172.21.15.23:0/1749271251 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c004300 con 0x7f1b2c107a90 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.129+0000 7f1b29d9b700 1 -- 172.21.15.23:0/1749271251 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b1c011b20 con 0x7f1b2c107a90 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 -- 172.21.15.23:0/1749271251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 msgr2=0x7f1b2c107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 --2- 172.21.15.23:0/1749271251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0098a0 tx=0x7f1b1c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 -- 172.21.15.23:0/1749271251 shutdown_connections 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 --2- 172.21.15.23:0/1749271251 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c107eb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 -- 172.21.15.23:0/1749271251 >> 172.21.15.23:0/1749271251 conn(0x7f1b2c1030d0 msgr2=0x7f1b2c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 -- 172.21.15.23:0/1749271251 shutdown_connections 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 -- 172.21.15.23:0/1749271251 wait complete. 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.130+0000 7f1b31093700 1 Processor -- start 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b31093700 1 -- start start 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b31093700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b31093700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c10c040 con 0x7f1b2c107a90 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:24.384 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b2ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33944/0 (socket says 172.21.15.23:33944) 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b2ad9d700 1 -- 172.21.15.23:0/1947134800 learned_addr learned my addr 172.21.15.23:0/1947134800 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b2ad9d700 1 -- 172.21.15.23:0/1947134800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c19bf60 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b2ad9d700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0116a0 tx=0x7f1b1c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.131+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b1c003a40 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c19c1f0 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b1c003be0 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1b1c01a720 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c19c650 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f1b1c01aa20 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.132+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c0637c0 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.133+0000 7f1b23fff700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 0x7f1b1803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.133+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1b1c04d4b0 con 0x7f1b2c107a90 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.133+0000 7f1b2a59c700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 0x7f1b1803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:24.385 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.133+0000 7f1b2a59c700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 0x7f1b1803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b14006fd0 tx=0x7f1b14006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.137+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1b1c011e20 con 0x7f1b2c107a90 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.348+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f1b2c19eee0 con 0x7f1b2c107a90 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.350+0000 7f1b23fff700 1 -- 172.21.15.23:0/1947134800 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f1b1c01adc0 con 0x7f1b2c107a90 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.352+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 msgr2=0x7f1b1803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.352+0000 7f1b31093700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 0x7f1b1803b390 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1b14006fd0 tx=0x7f1b14006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.352+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 msgr2=0x7f1b2c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.352+0000 7f1b31093700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c0116a0 tx=0x7f1b1c011ba0 comp rx=0 tx=0).stop 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 shutdown_connections 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f1b18038ed0 0x7f1b1803b390 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 --2- 172.21.15.23:0/1947134800 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1b2c107a90 0x7f1b2c19ba20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 >> 172.21.15.23:0/1947134800 conn(0x7f1b2c1030d0 msgr2=0x7f1b2c103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 shutdown_connections 2024-03-28T15:43:24.386 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:23.353+0000 7f1b31093700 1 -- 172.21.15.23:0/1947134800 wait complete. 2024-03-28T15:43:24.669 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:24 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1947134800' entity='client.admin' 2024-03-28T15:43:25.695 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:25 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:25.695 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:25 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/715118682' entity='client.admin' 2024-03-28T15:43:25.707 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.902+0000 7f10f8bce700 1 Processor -- start 2024-03-28T15:43:25.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.902+0000 7f10f8bce700 1 -- start start 2024-03-28T15:43:25.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f8bce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:25.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f8bce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4108950 con 0x7f10f4107f60 2024-03-28T15:43:25.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:25.708 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33952/0 (socket says 172.21.15.23:33952) 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f259c700 1 -- 172.21.15.23:0/775712008 learned_addr learned my addr 172.21.15.23:0/775712008 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.903+0000 7f10f259c700 1 -- 172.21.15.23:0/775712008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4108ad0 con 0x7f10f4107f60 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.904+0000 7f10f259c700 1 --2- 172.21.15.23:0/775712008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10e40098a0 tx=0x7f10e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d4ca621d15e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.904+0000 7f10f159a700 1 -- 172.21.15.23:0/775712008 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e4004160 con 0x7f10f4107f60 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.904+0000 7f10f159a700 1 -- 172.21.15.23:0/775712008 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e4004300 con 0x7f10f4107f60 2024-03-28T15:43:25.709 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.904+0000 7f10f159a700 1 -- 172.21.15.23:0/775712008 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e4011b20 con 0x7f10f4107f60 2024-03-28T15:43:25.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 -- 172.21.15.23:0/775712008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 msgr2=0x7f10f4108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:25.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 --2- 172.21.15.23:0/775712008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f10e40098a0 tx=0x7f10e4009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:25.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 -- 172.21.15.23:0/775712008 shutdown_connections 2024-03-28T15:43:25.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 --2- 172.21.15.23:0/775712008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f4108380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:25.710 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 -- 172.21.15.23:0/775712008 >> 172.21.15.23:0/775712008 conn(0x7f10f41016f0 msgr2=0x7f10f4103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 -- 172.21.15.23:0/775712008 shutdown_connections 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.905+0000 7f10f8bce700 1 -- 172.21.15.23:0/775712008 wait complete. 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f8bce700 1 Processor -- start 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f8bce700 1 -- start start 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f8bce700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:25.711 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f8bce700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f4106980 con 0x7f10f4107f60 2024-03-28T15:43:25.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:25.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f259c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33960/0 (socket says 172.21.15.23:33960) 2024-03-28T15:43:25.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f259c700 1 -- 172.21.15.23:0/715118682 learned_addr learned my addr 172.21.15.23:0/715118682 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:25.712 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.906+0000 7f10f259c700 1 -- 172.21.15.23:0/715118682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f419e060 con 0x7f10f4107f60 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10f259c700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10e40117e0 tx=0x7f10e4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e4003ba0 con 0x7f10f4107f60 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f419e350 con 0x7f10f4107f60 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10e401a910 con 0x7f10f4107f60 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10e40248f0 con 0x7f10f4107f60 2024-03-28T15:43:25.713 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.907+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f41a0f90 con 0x7f10f4107f60 2024-03-28T15:43:25.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.908+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f10e4022020 con 0x7f10f4107f60 2024-03-28T15:43:25.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.908+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f40637c0 con 0x7f10f4107f60 2024-03-28T15:43:25.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.908+0000 7f10e37fe700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 0x7f10dc03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:25.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.909+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f10e404e720 con 0x7f10f4107f60 2024-03-28T15:43:25.714 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.909+0000 7f10f1d9b700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 0x7f10dc03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.909+0000 7f10f1d9b700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 0x7f10dc03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006fd0 tx=0x7f10e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:24.914+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10e4018350 con 0x7f10f4107f60 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.120+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f10f41a1160 con 0x7f10f4107f60 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.123+0000 7f10e37fe700 1 -- 172.21.15.23:0/715118682 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f10f41a1160 con 0x7f10f4107f60 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 msgr2=0x7f10dc03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:25.715 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 0x7f10dc03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f10e8006fd0 tx=0x7f10e8006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:25.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 msgr2=0x7f10f419db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:25.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f10e40117e0 tx=0x7f10e4011ba0 comp rx=0 tx=0).stop 2024-03-28T15:43:25.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 shutdown_connections 2024-03-28T15:43:25.716 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f10dc038f20 0x7f10dc03b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:25.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 --2- 172.21.15.23:0/715118682 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10f4107f60 0x7f10f419db20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:25.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 >> 172.21.15.23:0/715118682 conn(0x7f10f41016f0 msgr2=0x7f10f4194140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:25.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 shutdown_connections 2024-03-28T15:43:25.717 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:25.126+0000 7f10f8bce700 1 -- 172.21.15.23:0/715118682 wait complete. 2024-03-28T15:43:25.717 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling the dashboard module... 2024-03-28T15:43:27.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:26 smithi023 ceph-mon[32448]: from='mgr.14120 172.21.15.23:0/1026770586' entity='mgr.x' 2024-03-28T15:43:27.039 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:26 smithi023 ceph-mon[32448]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:43:27.040 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:26 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/208482894' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-28T15:43:27.332 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setuser ceph since I am not root 2024-03-28T15:43:27.332 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setgroup ceph since I am not root 2024-03-28T15:43:27.935 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:27.797+0000 7f28ad631200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:43:27.936 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:27.894+0000 7f28ad631200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.146+0000 7f5eb180e700 1 Processor -- start 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.146+0000 7f5eb180e700 1 -- start start 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.146+0000 7f5eb180e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.146+0000 7f5eb180e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac07be50 con 0x7f5eac108560 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5eaaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5eaaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33968/0 (socket says 172.21.15.23:33968) 2024-03-28T15:43:27.944 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5eaaffd700 1 -- 172.21.15.23:0/3396245388 learned_addr learned my addr 172.21.15.23:0/3396245388 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5eaaffd700 1 -- 172.21.15.23:0/3396245388 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac10ae90 con 0x7f5eac108560 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5eaaffd700 1 --2- 172.21.15.23:0/3396245388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0098a0 tx=0x7f5e9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341c857d288e45e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.147+0000 7f5ea9ffb700 1 -- 172.21.15.23:0/3396245388 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e9c004160 con 0x7f5eac108560 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5ea9ffb700 1 -- 172.21.15.23:0/3396245388 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c004300 con 0x7f5eac108560 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5ea9ffb700 1 -- 172.21.15.23:0/3396245388 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e9c011b20 con 0x7f5eac108560 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 -- 172.21.15.23:0/3396245388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 msgr2=0x7f5eac10a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 --2- 172.21.15.23:0/3396245388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0098a0 tx=0x7f5e9c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 -- 172.21.15.23:0/3396245388 shutdown_connections 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 --2- 172.21.15.23:0/3396245388 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac10a950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 -- 172.21.15.23:0/3396245388 >> 172.21.15.23:0/3396245388 conn(0x7f5eac101f90 msgr2=0x7f5eac1043b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 -- 172.21.15.23:0/3396245388 shutdown_connections 2024-03-28T15:43:27.945 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.148+0000 7f5eb180e700 1 -- 172.21.15.23:0/3396245388 wait complete. 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eb180e700 1 Processor -- start 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eb180e700 1 -- start start 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eb180e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eb180e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eac10bef0 con 0x7f5eac108560 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eaaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eaaffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33978/0 (socket says 172.21.15.23:33978) 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.149+0000 7f5eaaffd700 1 -- 172.21.15.23:0/208482894 learned_addr learned my addr 172.21.15.23:0/208482894 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eaaffd700 1 -- 172.21.15.23:0/208482894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eac19bec0 con 0x7f5eac108560 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eaaffd700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0117e0 tx=0x7f5e9c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e9c003bf0 con 0x7f5eac108560 2024-03-28T15:43:27.946 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eac19c150 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e9c01a910 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e9c024970 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.150+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eac19c5b0 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.151+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eac0637c0 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.151+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5e9c022020 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.151+0000 7f5eb080c700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 0x7f5e9403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.151+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5e9c04d330 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.151+0000 7f5eaa7fc700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 0x7f5e9403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.152+0000 7f5eaa7fc700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 0x7f5e9403b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006fd0 tx=0x7f5ea0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.156+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5e9c011e20 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:26.435+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5eac19ef80 con 0x7f5eac108560 2024-03-28T15:43:27.947 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.213+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f5e9c00ba30 con 0x7f5eac108560 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.215+0000 7f5eb080c700 1 -- 172.21.15.23:0/208482894 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5e9c02d910 con 0x7f5eac108560 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 msgr2=0x7f5e9403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 0x7f5e9403b390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0006fd0 tx=0x7f5ea0006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 msgr2=0x7f5eac19b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e9c0117e0 tx=0x7f5e9c011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 shutdown_connections 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f5e94038ed0 0x7f5e9403b390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 --2- 172.21.15.23:0/208482894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5eac108560 0x7f5eac19b980 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 >> 172.21.15.23:0/208482894 conn(0x7f5eac101f90 msgr2=0x7f5eac102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 shutdown_connections 2024-03-28T15:43:27.948 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:27.217+0000 7f5eb180e700 1 -- 172.21.15.23:0/208482894 wait complete. 2024-03-28T15:43:28.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/208482894' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-28T15:43:28.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-mon[32448]: mgrmap e9: x(active, since 21s) 2024-03-28T15:43:28.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:28.571+0000 7f28ad631200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:43:28.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:28.671+0000 7f28ad631200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:43:28.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:28.818+0000 7f28ad631200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:43:29.215 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:28 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:28.921+0000 7f28ad631200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:43:29.215 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.007+0000 7f28ad631200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:43:29.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:43:29.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-28T15:43:29.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-28T15:43:29.226 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe9a4d7c700 1 Processor -- start 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe9a4d7c700 1 -- start start 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe9a4d7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe9a4d7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a007bdf0 con 0x7fe9a0106370 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33996/0 (socket says 172.21.15.23:33996) 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.366+0000 7fe99e59c700 1 -- 172.21.15.23:0/554206841 learned_addr learned my addr 172.21.15.23:0/554206841 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe99e59c700 1 -- 172.21.15.23:0/554206841 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a0106cd0 con 0x7fe9a0106370 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe99e59c700 1 --2- 172.21.15.23:0/554206841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9900098a0 tx=0x7fe990009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656229358977fa34 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe99d59a700 1 -- 172.21.15.23:0/554206841 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe990004160 con 0x7fe9a0106370 2024-03-28T15:43:29.227 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe99d59a700 1 -- 172.21.15.23:0/554206841 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe990004300 con 0x7fe9a0106370 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe99d59a700 1 -- 172.21.15.23:0/554206841 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe990011b20 con 0x7fe9a0106370 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/554206841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 msgr2=0x7fe9a0106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.367+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/554206841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe9900098a0 tx=0x7fe990009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/554206841 shutdown_connections 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/554206841 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a0106790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/554206841 >> 172.21.15.23:0/554206841 conn(0x7fe9a0101f70 msgr2=0x7fe9a0104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/554206841 shutdown_connections 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/554206841 wait complete. 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 Processor -- start 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.368+0000 7fe9a4d7c700 1 -- start start 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe9a4d7c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe9a4d7c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a0107da0 con 0x7fe9a0106370 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe99e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:34010/0 (socket says 172.21.15.23:34010) 2024-03-28T15:43:29.228 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe99e59c700 1 -- 172.21.15.23:0/4082132840 learned_addr learned my addr 172.21.15.23:0/4082132840 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe99e59c700 1 -- 172.21.15.23:0/4082132840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a019e0b0 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe99e59c700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe9900117e0 tx=0x7fe990011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe990003bf0 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a019e340 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe99001a910 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe990024970 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.369+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a019e7a0 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.370+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe990022020 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.371+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a00637c0 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.371+0000 7fe98f7fe700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 0x7fe98803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.371+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe99004d210 con 0x7fe9a0106370 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.371+0000 7fe99dd9b700 1 -- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 msgr2=0x7fe98803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.371+0000 7fe99dd9b700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 0x7fe98803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:43:29.229 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.376+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe990011e20 con 0x7fe9a0106370 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.572+0000 7fe99dd9b700 1 -- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 msgr2=0x7fe98803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.572+0000 7fe99dd9b700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 0x7fe98803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.642+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe9a0062da0 con 0x7fe9a0106370 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.643+0000 7fe98f7fe700 1 -- 172.21.15.23:0/4082132840 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe99000ba00 con 0x7fe9a0106370 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 msgr2=0x7fe98803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 0x7fe98803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 msgr2=0x7fe9a019db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe9900117e0 tx=0x7fe990011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 shutdown_connections 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7fe988038f20 0x7fe98803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 --2- 172.21.15.23:0/4082132840 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9a0106370 0x7fe9a019db70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 >> 172.21.15.23:0/4082132840 conn(0x7fe9a0101f70 msgr2=0x7fe9a01947e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:29.230 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 shutdown_connections 2024-03-28T15:43:29.231 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:28.645+0000 7fe9a4d7c700 1 -- 172.21.15.23:0/4082132840 wait complete. 2024-03-28T15:43:29.231 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for the mgr to restart... 2024-03-28T15:43:29.231 INFO:teuthology.orchestra.run.smithi023.stdout:Waiting for mgr epoch 9... 2024-03-28T15:43:29.492 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4082132840' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-28T15:43:29.492 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.286+0000 7f28ad631200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:43:29.492 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.400+0000 7f28ad631200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:43:29.492 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.491+0000 7f28ad631200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:43:29.773 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.592+0000 7f28ad631200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:43:29.774 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:29 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:29.773+0000 7f28ad631200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:43:31.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:30 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:30.761+0000 7f28ad631200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:43:31.028 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:30 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:30.896+0000 7f28ad631200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:43:31.330 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:31 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:31.238+0000 7f28ad631200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:43:31.331 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:31 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:31.330+0000 7f28ad631200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:43:31.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:31 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:31.533+0000 7f28ad631200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:43:32.020 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:31 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:31.725+0000 7f28ad631200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:43:32.282 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:32 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:32.020+0000 7f28ad631200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:43:32.283 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:32 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:32.282+0000 7f28ad631200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:43:32.629 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:32 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:32.528+0000 7f28ad631200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:43:33.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:33 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:33.362+0000 7f28ad631200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:43:34.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:33 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:33.944+0000 7f28ad631200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:43:34.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:34.046+0000 7f28ad631200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:43:34.272 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:43:34.272+0000 7f28ad631200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:43:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: Active manager daemon x restarted 2024-03-28T15:43:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: Activating manager daemon x 2024-03-28T15:43:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: osdmap e3: 0 total, 0 up, 0 in 2024-03-28T15:43:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: mgrmap e10: x(active, starting, since 0.00561306s) 2024-03-28T15:43:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: Manager daemon x is now available 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:34.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:35.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:43:35.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:43:35.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:35 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:35] ENGINE Bus STARTING 2024-03-28T15:43:35.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:35 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:35] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T15:43:35.417 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:35 smithi023 ceph-mon[32448]: mgrmap e11: x(active, since 1.01015s) 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout { 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout } 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.620+0000 7f7e5321d700 1 Processor -- start 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.620+0000 7f7e5321d700 1 -- start start 2024-03-28T15:43:36.311 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.620+0000 7f7e5321d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.620+0000 7f7e5321d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c07be30 con 0x7f7e4c108540 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e50fb9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e50fb9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44694/0 (socket says 172.21.15.23:44694) 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e50fb9700 1 -- 172.21.15.23:0/3751764815 learned_addr learned my addr 172.21.15.23:0/3751764815 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e50fb9700 1 -- 172.21.15.23:0/3751764815 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c10ae70 con 0x7f7e4c108540 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e50fb9700 1 --2- 172.21.15.23:0/3751764815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7e400098a0 tx=0x7f7e40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9f5fd8401d88faf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e4b7fe700 1 -- 172.21.15.23:0/3751764815 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40004160 con 0x7f7e4c108540 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e4b7fe700 1 -- 172.21.15.23:0/3751764815 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e40004300 con 0x7f7e4c108540 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.621+0000 7f7e4b7fe700 1 -- 172.21.15.23:0/3751764815 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40011b20 con 0x7f7e4c108540 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 -- 172.21.15.23:0/3751764815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 msgr2=0x7f7e4c10a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 --2- 172.21.15.23:0/3751764815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7e400098a0 tx=0x7f7e40009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 -- 172.21.15.23:0/3751764815 shutdown_connections 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 --2- 172.21.15.23:0/3751764815 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c10a930 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.312 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 -- 172.21.15.23:0/3751764815 >> 172.21.15.23:0/3751764815 conn(0x7f7e4c101f70 msgr2=0x7f7e4c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 -- 172.21.15.23:0/3751764815 shutdown_connections 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.622+0000 7f7e5321d700 1 -- 172.21.15.23:0/3751764815 wait complete. 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e5321d700 1 Processor -- start 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e5321d700 1 -- start start 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e5321d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e5321d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c10be80 con 0x7f7e4c108540 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e50fb9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e50fb9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44706/0 (socket says 172.21.15.23:44706) 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e50fb9700 1 -- 172.21.15.23:0/701912738 learned_addr learned my addr 172.21.15.23:0/701912738 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e50fb9700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c19beb0 con 0x7f7e4c108540 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.623+0000 7f7e50fb9700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e400117e0 tx=0x7f7e40011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.624+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e40003ba0 con 0x7f7e4c108540 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.624+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e4001a910 con 0x7f7e4c108540 2024-03-28T15:43:36.313 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.624+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c19c140 con 0x7f7e4c108540 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.624+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e400248f0 con 0x7f7e4c108540 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.624+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c19c5a0 con 0x7f7e4c108540 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7e40022020 con 0x7f7e4c108540 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e49ffb700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e3c03bb40 con 0x7f7e3c038f70 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7e4004d3c0 con 0x7f7e4c108540 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e4bfff700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.625+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.826+0000 7f7e4bfff700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:29.826+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:30.227+0000 7f7e4bfff700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:30.227+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:31.028+0000 7f7e4bfff700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:36.314 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:31.028+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:32.629+0000 7f7e4bfff700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/4208679988 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:32.629+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:34.289+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f7e4002d3f0 con 0x7f7e4c108540 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:34.289+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 msgr2=0x7f7e3c03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:34.289+0000 7f7e49ffb700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.292+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f7e4004f410 con 0x7f7e4c108540 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.292+0000 7f7e49ffb700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 0x7f7e3c03ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.292+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e3c03bb40 con 0x7f7e3c03c9f0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.293+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 0x7f7e3c03ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.293+0000 7f7e4bfff700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 0x7f7e3c03ede0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7e38003a10 tx=0x7f7e380092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.294+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f7e3c03bb40 con 0x7f7e3c03c9f0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7e4c0637c0 con 0x7f7e3c03c9f0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e49ffb700 1 -- 172.21.15.23:0/701912738 <== mgr.14152 v2:172.21.15.23:6800/1340727098 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7e4c0637c0 con 0x7f7e3c03c9f0 2024-03-28T15:43:36.315 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 msgr2=0x7f7e3c03ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 0x7f7e3c03ede0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7e38003a10 tx=0x7f7e380092b0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 msgr2=0x7f7e4c19b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7e400117e0 tx=0x7f7e40011ba0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 shutdown_connections 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7e4c108540 0x7f7e4c19b970 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7e3c03c9f0 0x7f7e3c03ede0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 --2- 172.21.15.23:0/701912738 >> [v2:172.21.15.23:6800/4208679988,v1:172.21.15.23:6801/4208679988] conn(0x7f7e3c038f70 0x7f7e3c03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 >> 172.21.15.23:0/701912738 conn(0x7f7e4c101f70 msgr2=0x7f7e4c1029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 shutdown_connections 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:35.301+0000 7f7e5321d700 1 -- 172.21.15.23:0/701912738 wait complete. 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:mgr epoch 9 is available 2024-03-28T15:43:36.316 INFO:teuthology.orchestra.run.smithi023.stdout:Generating a dashboard self-signed certificate... 2024-03-28T15:43:37.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:36 smithi023 ceph-mon[32448]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-28T15:43:37.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:36 smithi023 ceph-mon[32448]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-28T15:43:37.087 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:36 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:35] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T15:43:37.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:36 smithi023 ceph-mon[32448]: [28/Mar/2024:15:43:35] ENGINE Bus STARTED 2024-03-28T15:43:37.088 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f452a9d4700 1 Processor -- start 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f452a9d4700 1 -- start start 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f452a9d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f452a9d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524108a00 con 0x7f4524108010 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f4523fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f4523fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44800/0 (socket says 172.21.15.23:44800) 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.722+0000 7f4523fff700 1 -- 172.21.15.23:0/57410156 learned_addr learned my addr 172.21.15.23:0/57410156 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.723+0000 7f4523fff700 1 -- 172.21.15.23:0/57410156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524105b10 con 0x7f4524108010 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.723+0000 7f4523fff700 1 --2- 172.21.15.23:0/57410156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45140098a0 tx=0x7f4514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c476fc30ad751188 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:37.702 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.723+0000 7f4522ffd700 1 -- 172.21.15.23:0/57410156 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4514004160 con 0x7f4524108010 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.723+0000 7f4522ffd700 1 -- 172.21.15.23:0/57410156 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4514004300 con 0x7f4524108010 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.723+0000 7f4522ffd700 1 -- 172.21.15.23:0/57410156 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4514011b20 con 0x7f4524108010 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- 172.21.15.23:0/57410156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 msgr2=0x7f4524108430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 --2- 172.21.15.23:0/57410156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f45140098a0 tx=0x7f4514009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- 172.21.15.23:0/57410156 shutdown_connections 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 --2- 172.21.15.23:0/57410156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f4524108430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- 172.21.15.23:0/57410156 >> 172.21.15.23:0/57410156 conn(0x7f45241016f0 msgr2=0x7f4524103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- 172.21.15.23:0/57410156 shutdown_connections 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- 172.21.15.23:0/57410156 wait complete. 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 Processor -- start 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.724+0000 7f452a9d4700 1 -- start start 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f452a9d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f452a9d4700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524106bd0 con 0x7f4524108010 2024-03-28T15:43:37.703 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f4523fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f4523fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44812/0 (socket says 172.21.15.23:44812) 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f4523fff700 1 -- 172.21.15.23:0/1810699404 learned_addr learned my addr 172.21.15.23:0/1810699404 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f4523fff700 1 -- 172.21.15.23:0/1810699404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f452419e0c0 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.725+0000 7f4523fff700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f45140117e0 tx=0x7f4514011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4514003bf0 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452419e350 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f451401a910 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4514024970 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452419e7b0 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.726+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f4514022020 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.727+0000 7f45217fa700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 0x7f450c03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.727+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45240637c0 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.727+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f451404d100 con 0x7f4524108010 2024-03-28T15:43:37.704 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.727+0000 7f45237fe700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 0x7f450c03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.727+0000 7f45237fe700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 0x7f450c03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4518006fd0 tx=0x7f4518006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.732+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4514011e20 con 0x7f4524108010 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.944+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f45240625d0 con 0x7f450c038e10 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:36.961+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f451402d910 con 0x7f4524108010 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.087+0000 7f45217fa700 1 -- 172.21.15.23:0/1810699404 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f45240625d0 con 0x7f450c038e10 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 msgr2=0x7f450c03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 0x7f450c03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4518006fd0 tx=0x7f4518006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 msgr2=0x7f452419db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f45140117e0 tx=0x7f4514011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 shutdown_connections 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f450c038e10 0x7f450c03b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 --2- 172.21.15.23:0/1810699404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4524108010 0x7f452419db80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.089+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 >> 172.21.15.23:0/1810699404 conn(0x7f45241016f0 msgr2=0x7f4524194260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.090+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 shutdown_connections 2024-03-28T15:43:37.705 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:37.090+0000 7f452a9d4700 1 -- 172.21.15.23:0/1810699404 wait complete. 2024-03-28T15:43:37.706 INFO:teuthology.orchestra.run.smithi023.stdout:Creating initial admin user... 2024-03-28T15:43:38.265 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:37 smithi023 ceph-mon[32448]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:38.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:37 smithi023 ceph-mon[32448]: mgrmap e12: x(active, since 2s) 2024-03-28T15:43:38.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:38.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:38.266 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:39.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:39 smithi023 ceph-mon[32448]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:39.788 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3Wfb2sb45XeKl3t26BjWP.nimInZoBCY6vu8aASaS1Alp6ERshCH6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711640618, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.114+0000 7f38a0878700 1 Processor -- start 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.114+0000 7f38a0878700 1 -- start start 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.114+0000 7f38a0878700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.114+0000 7f38a0878700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c07be90 con 0x7f389c1084e0 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44818/0 (socket says 172.21.15.23:44818) 2024-03-28T15:43:39.797 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389a59c700 1 -- 172.21.15.23:0/2332213882 learned_addr learned my addr 172.21.15.23:0/2332213882 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389a59c700 1 -- 172.21.15.23:0/2332213882 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c10ae10 con 0x7f389c1084e0 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389a59c700 1 --2- 172.21.15.23:0/2332213882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f388c0098a0 tx=0x7f388c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f937666966d6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389959a700 1 -- 172.21.15.23:0/2332213882 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f388c004160 con 0x7f389c1084e0 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389959a700 1 -- 172.21.15.23:0/2332213882 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c004300 con 0x7f389c1084e0 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.115+0000 7f389959a700 1 -- 172.21.15.23:0/2332213882 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f388c011b20 con 0x7f389c1084e0 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.116+0000 7f38a0878700 1 -- 172.21.15.23:0/2332213882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 msgr2=0x7f389c10a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.116+0000 7f38a0878700 1 --2- 172.21.15.23:0/2332213882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f388c0098a0 tx=0x7f388c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.116+0000 7f38a0878700 1 -- 172.21.15.23:0/2332213882 shutdown_connections 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.116+0000 7f38a0878700 1 --2- 172.21.15.23:0/2332213882 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c10a8d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.116+0000 7f38a0878700 1 -- 172.21.15.23:0/2332213882 >> 172.21.15.23:0/2332213882 conn(0x7f389c101f70 msgr2=0x7f389c104390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 -- 172.21.15.23:0/2332213882 shutdown_connections 2024-03-28T15:43:39.798 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 -- 172.21.15.23:0/2332213882 wait complete. 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 Processor -- start 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 -- start start 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.117+0000 7f38a0878700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c10be80 con 0x7f389c1084e0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f389a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f389a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44834/0 (socket says 172.21.15.23:44834) 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f389a59c700 1 -- 172.21.15.23:0/483459914 learned_addr learned my addr 172.21.15.23:0/483459914 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f389a59c700 1 -- 172.21.15.23:0/483459914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389c1082a0 con 0x7f389c1084e0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f389a59c700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f388c0117e0 tx=0x7f388c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f388c003bf0 con 0x7f389c1084e0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c01a910 con 0x7f389c1084e0 2024-03-28T15:43:39.799 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f389c106490 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f388c024970 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.118+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c106950 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.119+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f388c022020 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.119+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f389c0637c0 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.119+0000 7f388b7fe700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 0x7f388403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.119+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f388c04d360 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.120+0000 7f3899d9b700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 0x7f388403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.120+0000 7f3899d9b700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 0x7f388403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3890006fd0 tx=0x7f3890006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.125+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f388c011e20 con 0x7f389c1084e0 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.363+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f389c1070c0 con 0x7f3884038f20 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.609+0000 7f388b7fe700 1 -- 172.21.15.23:0/483459914 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f389c1070c0 con 0x7f3884038f20 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 msgr2=0x7f388403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:39.800 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 0x7f388403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3890006fd0 tx=0x7f3890006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 msgr2=0x7f389c107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f388c0117e0 tx=0x7f388c011c10 comp rx=0 tx=0).stop 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 shutdown_connections 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3884038f20 0x7f388403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 --2- 172.21.15.23:0/483459914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f389c1084e0 0x7f389c107d60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 >> 172.21.15.23:0/483459914 conn(0x7f389c101f70 msgr2=0x7f389c102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 shutdown_connections 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:38.612+0000 7f38a0878700 1 -- 172.21.15.23:0/483459914 wait complete. 2024-03-28T15:43:39.801 INFO:teuthology.orchestra.run.smithi023.stdout:Fetching dashboard port number... 2024-03-28T15:43:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:40 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3231361547' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-28T15:43:40.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stdout 8443 2024-03-28T15:43:40.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f9370e06700 1 Processor -- start 2024-03-28T15:43:40.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f9370e06700 1 -- start start 2024-03-28T15:43:40.952 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f9370e06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f9370e06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c108480 con 0x7f936c107a90 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f936a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f936a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55196/0 (socket says 172.21.15.23:55196) 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.201+0000 7f936a59c700 1 -- 172.21.15.23:0/249111865 learned_addr learned my addr 172.21.15.23:0/249111865 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.202+0000 7f936a59c700 1 -- 172.21.15.23:0/249111865 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c108c90 con 0x7f936c107a90 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.202+0000 7f936a59c700 1 --2- 172.21.15.23:0/249111865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f935c0098a0 tx=0x7f935c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf50021f3e398d09 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.202+0000 7f9369d9b700 1 -- 172.21.15.23:0/249111865 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f935c004160 con 0x7f936c107a90 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.202+0000 7f9369d9b700 1 -- 172.21.15.23:0/249111865 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935c004300 con 0x7f936c107a90 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.202+0000 7f9369d9b700 1 -- 172.21.15.23:0/249111865 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f935c011b20 con 0x7f936c107a90 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 -- 172.21.15.23:0/249111865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 msgr2=0x7f936c107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 --2- 172.21.15.23:0/249111865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f935c0098a0 tx=0x7f935c009bb0 comp rx=0 tx=0).stop 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 -- 172.21.15.23:0/249111865 shutdown_connections 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 --2- 172.21.15.23:0/249111865 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c107eb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:40.953 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 -- 172.21.15.23:0/249111865 >> 172.21.15.23:0/249111865 conn(0x7f936c1030d0 msgr2=0x7f936c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 -- 172.21.15.23:0/249111865 shutdown_connections 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.203+0000 7f9370e06700 1 -- 172.21.15.23:0/249111865 wait complete. 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f9370e06700 1 Processor -- start 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f9370e06700 1 -- start start 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f9370e06700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f9370e06700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c10c040 con 0x7f936c107a90 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f936a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f936a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55208/0 (socket says 172.21.15.23:55208) 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f936a59c700 1 -- 172.21.15.23:0/3231361547 learned_addr learned my addr 172.21.15.23:0/3231361547 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f936a59c700 1 -- 172.21.15.23:0/3231361547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c19bf60 con 0x7f936c107a90 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f936a59c700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f935c0116a0 tx=0x7f935c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f935c003a40 con 0x7f936c107a90 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935c003be0 con 0x7f936c107a90 2024-03-28T15:43:40.954 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c19c1f0 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.204+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f935c01a720 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.205+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c19c650 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.205+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f935c01aa40 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.205+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c0637c0 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.206+0000 7f93637fe700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 0x7f935803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.206+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f935c04d5e0 con 0x7f936c107a90 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.206+0000 7f9363fff700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 0x7f935803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.206+0000 7f9363fff700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 0x7f935803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:40.955 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.210+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f935c011e20 con 0x7f936c107a90 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.414+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f936c19f0e0 con 0x7f936c107a90 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.414+0000 7f93637fe700 1 -- 172.21.15.23:0/3231361547 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f935c00bc70 con 0x7f936c107a90 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 msgr2=0x7f935803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 0x7f935803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9354006fd0 tx=0x7f9354006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 msgr2=0x7f936c19ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f935c0116a0 tx=0x7f935c011ba0 comp rx=0 tx=0).stop 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 shutdown_connections 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9358038f20 0x7f935803b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 --2- 172.21.15.23:0/3231361547 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f936c107a90 0x7f936c19ba20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 >> 172.21.15.23:0/3231361547 conn(0x7f936c1030d0 msgr2=0x7f936c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 shutdown_connections 2024-03-28T15:43:40.956 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:40.417+0000 7f9370e06700 1 -- 172.21.15.23:0/3231361547 wait complete. 2024-03-28T15:43:40.958 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-28T15:43:40.958 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout disabled 2024-03-28T15:43:40.966 INFO:teuthology.orchestra.run.smithi023.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-28T15:43:40.966 INFO:teuthology.orchestra.run.smithi023.stdout:systemctl: stdout inactive 2024-03-28T15:43:40.966 INFO:teuthology.orchestra.run.smithi023.stdout:firewalld.service is not enabled 2024-03-28T15:43:40.966 INFO:teuthology.orchestra.run.smithi023.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-28T15:43:40.967 INFO:teuthology.orchestra.run.smithi023.stdout:Ceph Dashboard is now available at: 2024-03-28T15:43:40.967 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout: URL: https://smithi023.front.sepia.ceph.com:8443/ 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout: User: admin 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout: Password: 8ns48av45s 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout:Saving cluster configuration to /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config directory 2024-03-28T15:43:40.968 INFO:teuthology.orchestra.run.smithi023.stdout:Enabling autotune for osd_memory_target 2024-03-28T15:43:41.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:41 smithi023 ceph-mon[32448]: mgrmap e13: x(active, since 6s) 2024-03-28T15:43:42.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb061590700 1 Processor -- start 2024-03-28T15:43:42.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb061590700 1 -- start start 2024-03-28T15:43:42.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb061590700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:42.169 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb061590700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c108460 con 0x7fb05c107a70 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb05affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb05affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55216/0 (socket says 172.21.15.23:55216) 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.417+0000 7fb05affd700 1 -- 172.21.15.23:0/3580984762 learned_addr learned my addr 172.21.15.23:0/3580984762 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb05affd700 1 -- 172.21.15.23:0/3580984762 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c108c70 con 0x7fb05c107a70 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb05affd700 1 --2- 172.21.15.23:0/3580984762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb054009e80 tx=0x7fb05400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab1c16d6d5eb840 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb059ffb700 1 -- 172.21.15.23:0/3580984762 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb054017070 con 0x7fb05c107a70 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb059ffb700 1 -- 172.21.15.23:0/3580984762 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb05400b8c0 con 0x7fb05c107a70 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb059ffb700 1 -- 172.21.15.23:0/3580984762 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0540045c0 con 0x7fb05c107a70 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb061590700 1 -- 172.21.15.23:0/3580984762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 msgr2=0x7fb05c107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb061590700 1 --2- 172.21.15.23:0/3580984762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb054009e80 tx=0x7fb05400b330 comp rx=0 tx=0).stop 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb061590700 1 -- 172.21.15.23:0/3580984762 shutdown_connections 2024-03-28T15:43:42.170 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb061590700 1 --2- 172.21.15.23:0/3580984762 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c107e90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.418+0000 7fb061590700 1 -- 172.21.15.23:0/3580984762 >> 172.21.15.23:0/3580984762 conn(0x7fb05c1030d0 msgr2=0x7fb05c1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 -- 172.21.15.23:0/3580984762 shutdown_connections 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 -- 172.21.15.23:0/3580984762 wait complete. 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 Processor -- start 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 -- start start 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.419+0000 7fb061590700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c10c020 con 0x7fb05c107a70 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb05affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb05affd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55220/0 (socket says 172.21.15.23:55220) 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb05affd700 1 -- 172.21.15.23:0/3815711715 learned_addr learned my addr 172.21.15.23:0/3815711715 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb05affd700 1 -- 172.21.15.23:0/3815711715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c19e140 con 0x7fb05c107a70 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb05affd700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb05400bca0 tx=0x7fb05400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb054020020 con 0x7fb05c107a70 2024-03-28T15:43:42.171 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb054003d90 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c19e3d0 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb054017040 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.420+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c1a10c0 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.421+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb054023020 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.421+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c0637c0 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.421+0000 7fb053fff700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 0x7fb04803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.421+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb05400f050 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.422+0000 7fb05a7fc700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 0x7fb04803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.422+0000 7fb05a7fc700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 0x7fb04803b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb044006fd0 tx=0x7fb044006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.426+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb05402a020 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.631+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb05c1a1290 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.632+0000 7fb053fff700 1 -- 172.21.15.23:0/3815711715 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb05c1a1290 con 0x7fb05c107a70 2024-03-28T15:43:42.172 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 msgr2=0x7fb04803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 0x7fb04803b480 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb044006fd0 tx=0x7fb044006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 msgr2=0x7fb05c19dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb05400bca0 tx=0x7fb05400bd80 comp rx=0 tx=0).stop 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 shutdown_connections 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb048038fc0 0x7fb04803b480 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 --2- 172.21.15.23:0/3815711715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb05c107a70 0x7fb05c19dc00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 >> 172.21.15.23:0/3815711715 conn(0x7fb05c1030d0 msgr2=0x7fb05c103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 shutdown_connections 2024-03-28T15:43:42.173 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:41.634+0000 7fb061590700 1 -- 172.21.15.23:0/3815711715 wait complete. 2024-03-28T15:43:43.417 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f1b03e700 1 Processor -- start 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f1b03e700 1 -- start start 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f1b03e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f1b03e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1407be90 con 0x7f5f141084c0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f18dda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f18dda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55232/0 (socket says 172.21.15.23:55232) 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.575+0000 7f5f18dda700 1 -- 172.21.15.23:0/2203999465 learned_addr learned my addr 172.21.15.23:0/2203999465 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f18dda700 1 -- 172.21.15.23:0/2203999465 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1410adf0 con 0x7f5f141084c0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f18dda700 1 --2- 172.21.15.23:0/2203999465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009e80 tx=0x7f5f0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133443c2cbc8b4b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f137fe700 1 -- 172.21.15.23:0/2203999465 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f0c017070 con 0x7f5f141084c0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f137fe700 1 -- 172.21.15.23:0/2203999465 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f0c00b8c0 con 0x7f5f141084c0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f137fe700 1 -- 172.21.15.23:0/2203999465 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f0c0045c0 con 0x7f5f141084c0 2024-03-28T15:43:43.418 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.576+0000 7f5f1b03e700 1 -- 172.21.15.23:0/2203999465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 msgr2=0x7f5f1410a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/2203999465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009e80 tx=0x7f5f0c00b330 comp rx=0 tx=0).stop 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- 172.21.15.23:0/2203999465 shutdown_connections 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/2203999465 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f1410a8b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- 172.21.15.23:0/2203999465 >> 172.21.15.23:0/2203999465 conn(0x7f5f14101f10 msgr2=0x7f5f14104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- 172.21.15.23:0/2203999465 shutdown_connections 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- 172.21.15.23:0/2203999465 wait complete. 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 Processor -- start 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- start start 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f1b03e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1410be60 con 0x7f5f141084c0 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f18dda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f18dda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55238/0 (socket says 172.21.15.23:55238) 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.577+0000 7f5f18dda700 1 -- 172.21.15.23:0/3440676607 learned_addr learned my addr 172.21.15.23:0/3440676607 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f18dda700 1 -- 172.21.15.23:0/3440676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f14108280 con 0x7f5f141084c0 2024-03-28T15:43:43.419 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f18dda700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00b7b0 tx=0x7f5f0c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f0c020020 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f0c003d90 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f14106470 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f0c017040 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.578+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f141069b0 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.579+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f5f0c023020 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.579+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f140637c0 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.579+0000 7f5f11ffb700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 0x7f5efc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.579+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f0c00f050 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.579+0000 7f5f13fff700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 0x7f5efc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.580+0000 7f5f13fff700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 0x7f5efc03b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f08006fd0 tx=0x7f5f08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.584+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f0c02a020 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.902+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f5f14106b80 con 0x7f5f141084c0 2024-03-28T15:43:43.420 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.909+0000 7f5f11ffb700 1 -- 172.21.15.23:0/3440676607 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f5f0c01b730 con 0x7f5f141084c0 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.911+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 msgr2=0x7f5efc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.911+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 0x7f5efc03b4e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f08006fd0 tx=0x7f5f08006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.911+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 msgr2=0x7f5f14107d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.911+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00b7b0 tx=0x7f5f0c00bd50 comp rx=0 tx=0).stop 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 shutdown_connections 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5efc039020 0x7f5efc03b4e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 --2- 172.21.15.23:0/3440676607 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f141084c0 0x7f5f14107d40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 >> 172.21.15.23:0/3440676607 conn(0x7f5f14101f10 msgr2=0x7f5f141029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 shutdown_connections 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr 2024-03-28T15:43:42.912+0000 7f5f1b03e700 1 -- 172.21.15.23:0/3440676607 wait complete. 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.421 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout:Or, if you are only running a single cluster on this host: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: ceph telemetry on 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout:For more information see: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:43.422 INFO:teuthology.orchestra.run.smithi023.stdout:Bootstrap complete. 2024-03-28T15:43:43.456 INFO:tasks.cephadm:Fetching config... 2024-03-28T15:43:43.456 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:43:43.456 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-28T15:43:43.471 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-28T15:43:43.471 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:43:43.471 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-28T15:43:43.526 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-28T15:43:43.526 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:43:43.526 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-28T15:43:43.593 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-28T15:43:43.593 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:43:43.593 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-28T15:43:43.647 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-28T15:43:43.647 DEBUG:teuthology.orchestra.run.smithi023:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:43:43.726 INFO:teuthology.orchestra.run.smithi023.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:43:43.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:43:43.797 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:43:43.812 DEBUG:teuthology.orchestra.run.smithi120:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-28T15:43:43.860 INFO:teuthology.orchestra.run.smithi120.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDKGbQZpuzy5qjG3qXpb4VZdIFx1m9MyKEhjdmGrUP9pMGfrJ1eSPoi9h2qI0WpB87N1xqf5NK3njs0Sq3wptsGgwEEqIrV6XJD8ysZvt7m8nHe6Vhin3UValyy7i4QBJTdTGjcHVYUAQGVId8MBO5iWGc7uupQDZYQx7zrnt+VtRpqWSQtH+/F+gT0mOaBFBB3NWbGJU42uP5pOpVuRVsiSltI2mzJMstxLGtfImf3WCtkYsJTxUwec/g/lbPJl5XUQEB8K85xxKia/qavcPCyynAfFvyrwPPSMll7NMKVUGyMWHBpLggUfPyJ62wRtrsWiwV1FomoQtpxXRrXQkOrC8B7Q5bFLBUdki72od/AdzLcoyOj7AzWhuq5WaSOTgmi8IwrCOQwHAv/ibWrAFXx43rHoajJsFcbSSbTyERY2E+70tsOxC+VH+yaBMWqZYAcU/bfVpdOjD/8HLsNBZhVC3tGZ9gL99V3JMlKIzB/YYHULqom5NVU2ly1yHXKsuE= ceph-a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:43:43.877 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-28T15:43:44.120 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:43:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:43 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3440676607' entity='client.admin' 2024-03-28T15:43:46.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:46.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:46.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi023", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:43:46.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:46.738 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:46.739 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 -- 172.21.15.23:0/1655314455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 msgr2=0x7ffb50102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1655314455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50102010 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb40009910 tx=0x7ffb40009c20 comp rx=0 tx=0).stop 2024-03-28T15:43:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 -- 172.21.15.23:0/1655314455 shutdown_connections 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1655314455 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50102010 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 -- 172.21.15.23:0/1655314455 >> 172.21.15.23:0/1655314455 conn(0x7ffb500fd1b0 msgr2=0x7ffb500ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 -- 172.21.15.23:0/1655314455 shutdown_connections 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.844+0000 7ffb553ef700 1 -- 172.21.15.23:0/1655314455 wait complete. 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb553ef700 1 Processor -- start 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb553ef700 1 -- start start 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb553ef700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb553ef700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb50107c80 con 0x7ffb50101bf0 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb4effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:46.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb4effd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55246/0 (socket says 172.21.15.23:55246) 2024-03-28T15:43:46.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb4effd700 1 -- 172.21.15.23:0/1225652450 learned_addr learned my addr 172.21.15.23:0/1225652450 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:46.847 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.845+0000 7ffb4effd700 1 -- 172.21.15.23:0/1225652450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50076390 con 0x7ffb50101bf0 2024-03-28T15:43:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb4effd700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb40006010 tx=0x7ffb40005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb40027070 con 0x7ffb50101bf0 2024-03-28T15:43:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb4000be40 con 0x7ffb50101bf0 2024-03-28T15:43:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb50074580 con 0x7ffb50101bf0 2024-03-28T15:43:46.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb50074ac0 con 0x7ffb50101bf0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.846+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffb4000f460 con 0x7ffb50101bf0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.847+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ffb4002a070 con 0x7ffb50101bf0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.847+0000 7ffb37fff700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 0x7ffb3803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.847+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb50108400 con 0x7ffb50101bf0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.847+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffb4001e030 con 0x7ffb50101bf0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.847+0000 7ffb4e7fc700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 0x7ffb3803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:46.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.848+0000 7ffb4e7fc700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 0x7ffb3803b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffb44006fd0 tx=0x7ffb44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:46.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:46.852+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffb40031030 con 0x7ffb50101bf0 2024-03-28T15:43:47.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.060+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7ffb501906c0 con 0x7ffb50101bf0 2024-03-28T15:43:47.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.063+0000 7ffb37fff700 1 -- 172.21.15.23:0/1225652450 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7ffb40013070 con 0x7ffb50101bf0 2024-03-28T15:43:47.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 msgr2=0x7ffb3803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 0x7ffb3803b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ffb44006fd0 tx=0x7ffb44006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 msgr2=0x7ffb50075e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffb40006010 tx=0x7ffb40005050 comp rx=0 tx=0).stop 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 shutdown_connections 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffb38038fc0 0x7ffb3803b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.065+0000 7ffb553ef700 1 --2- 172.21.15.23:0/1225652450 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffb50101bf0 0x7ffb50075e50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.066+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 >> 172.21.15.23:0/1225652450 conn(0x7ffb500fd1b0 msgr2=0x7ffb500fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:47.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.066+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 shutdown_connections 2024-03-28T15:43:47.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:47.066+0000 7ffb553ef700 1 -- 172.21.15.23:0/1225652450 wait complete. 2024-03-28T15:43:48.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1225652450' entity='client.admin' 2024-03-28T15:43:48.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:48.161 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:48.191 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-28T15:43:48.191 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-28T15:43:48.432 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:43:50.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 -- 172.21.15.23:0/1671769233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 msgr2=0x7f2bc0103610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:50.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 --2- 172.21.15.23:0/1671769233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc0103610 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2bb800b0a0 tx=0x7f2bb800b3b0 comp rx=0 tx=0).stop 2024-03-28T15:43:50.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 -- 172.21.15.23:0/1671769233 shutdown_connections 2024-03-28T15:43:50.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 --2- 172.21.15.23:0/1671769233 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc0103610 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:50.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 -- 172.21.15.23:0/1671769233 >> 172.21.15.23:0/1671769233 conn(0x7f2bc00fb7f0 msgr2=0x7f2bc00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 -- 172.21.15.23:0/1671769233 shutdown_connections 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.998+0000 7f2bc713f700 1 -- 172.21.15.23:0/1671769233 wait complete. 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc713f700 1 Processor -- start 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc713f700 1 -- start start 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc713f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc713f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc0107c40 con 0x7f2bc01031f0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc4edb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc4edb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47986/0 (socket says 172.21.15.23:47986) 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc4edb700 1 -- 172.21.15.23:0/817451146 learned_addr learned my addr 172.21.15.23:0/817451146 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc4edb700 1 -- 172.21.15.23:0/817451146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc0197720 con 0x7f2bc01031f0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc4edb700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8015040 tx=0x7f2bb8009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bb8017040 con 0x7f2bc01031f0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bb8022070 con 0x7f2bc01031f0 2024-03-28T15:43:50.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc01979b0 con 0x7f2bc01031f0 2024-03-28T15:43:50.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:49.999+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2bb801d5b0 con 0x7f2bc01031f0 2024-03-28T15:43:50.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.000+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc019a6a0 con 0x7f2bc01031f0 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.000+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2bb801d8b0 con 0x7f2bc01031f0 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.000+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc01083c0 con 0x7f2bc01031f0 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.001+0000 7f2bbdffb700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 0x7f2ba803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.001+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2bb800f050 con 0x7f2bc01031f0 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.001+0000 7f2bbffff700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 0x7f2ba803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:50.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.001+0000 7f2bbffff700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 0x7f2ba803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4006fd0 tx=0x7f2bb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:50.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.006+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2bb8007e20 con 0x7f2bc01031f0 2024-03-28T15:43:50.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.221+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc002db70 con 0x7f2ba8038fc0 2024-03-28T15:43:50.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.224+0000 7f2bbdffb700 1 -- 172.21.15.23:0/817451146 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2bc002db70 con 0x7f2ba8038fc0 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 msgr2=0x7f2ba803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 0x7f2ba803b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4006fd0 tx=0x7f2bb4006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 msgr2=0x7f2bc01971e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8015040 tx=0x7f2bb8009790 comp rx=0 tx=0).stop 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 shutdown_connections 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2ba8038fc0 0x7f2ba803b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 --2- 172.21.15.23:0/817451146 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2bc01031f0 0x7f2bc01971e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 >> 172.21.15.23:0/817451146 conn(0x7f2bc00fb7f0 msgr2=0x7f2bc00fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 shutdown_connections 2024-03-28T15:43:50.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:50.227+0000 7f2bc713f700 1 -- 172.21.15.23:0/817451146 wait complete. 2024-03-28T15:43:50.996 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi117 2024-03-28T15:43:50.997 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:43:50.997 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.conf 2024-03-28T15:43:51.015 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:43:51.015 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:43:51.072 INFO:tasks.cephadm:Adding host smithi117 to orchestrator... 2024-03-28T15:43:51.072 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch host add smithi117 2024-03-28T15:43:51.309 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:43:51.532 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:51.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:52.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:52 smithi023 ceph-mon[32448]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:43:52.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:52 smithi023 ceph-mon[32448]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:43:52.484 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:52 smithi023 ceph-mon[32448]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 -- 172.21.15.23:0/2441121976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 msgr2=0x7f9d681021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 --2- 172.21.15.23:0/2441121976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d681021b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d6000b0a0 tx=0x7f9d6000b3b0 comp rx=0 tx=0).stop 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 -- 172.21.15.23:0/2441121976 shutdown_connections 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 --2- 172.21.15.23:0/2441121976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d681021b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 -- 172.21.15.23:0/2441121976 >> 172.21.15.23:0/2441121976 conn(0x7f9d680fd6a0 msgr2=0x7f9d680ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 -- 172.21.15.23:0/2441121976 shutdown_connections 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 -- 172.21.15.23:0/2441121976 wait complete. 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.809+0000 7f9d70296700 1 Processor -- start 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d70296700 1 -- start start 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d70296700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:52.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d70296700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6810c1a0 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d6e032700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d6e032700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47996/0 (socket says 172.21.15.23:47996) 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d6e032700 1 -- 172.21.15.23:0/2513489772 learned_addr learned my addr 172.21.15.23:0/2513489772 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d6e032700 1 -- 172.21.15.23:0/2513489772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68197bd0 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d6e032700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9d60015040 tx=0x7f9d60003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d60017040 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d60022070 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d68197e60 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d6001d5d0 con 0x7f9d68101dd0 2024-03-28T15:43:52.812 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.810+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6819bf60 con 0x7f9d68101dd0 2024-03-28T15:43:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.811+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9d60004000 con 0x7f9d68101dd0 2024-03-28T15:43:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.811+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d681032b0 con 0x7f9d68101dd0 2024-03-28T15:43:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.811+0000 7f9d677fe700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 0x7f9d5003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.812+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d6000f050 con 0x7f9d68101dd0 2024-03-28T15:43:52.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.812+0000 7f9d6d831700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 0x7f9d5003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:52.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.812+0000 7f9d6d831700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 0x7f9d5003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:52.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:52.816+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d60007e20 con 0x7f9d68101dd0 2024-03-28T15:43:53.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:53.024+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}) v1 -- 0x7f9d680637c0 con 0x7f9d50038fc0 2024-03-28T15:43:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:53 smithi023 ceph-mon[32448]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:43:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:54 smithi023 ceph-mon[32448]: Deploying cephadm binary to smithi117 2024-03-28T15:43:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:55 smithi023 ceph-mon[32448]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:43:57.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.286+0000 7f9d677fe700 1 -- 172.21.15.23:0/2513489772 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9d680637c0 con 0x7f9d50038fc0 2024-03-28T15:43:57.287 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi117' with addr '172.21.15.117' 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 msgr2=0x7f9d5003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 0x7f9d5003b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006fd0 tx=0x7f9d5c006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 msgr2=0x7f9d68197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9d60015040 tx=0x7f9d60003d50 comp rx=0 tx=0).stop 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 shutdown_connections 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9d50038fc0 0x7f9d5003b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 --2- 172.21.15.23:0/2513489772 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9d68101dd0 0x7f9d68197690 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.288+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 >> 172.21.15.23:0/2513489772 conn(0x7f9d680fd6a0 msgr2=0x7f9d681037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.289+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 shutdown_connections 2024-03-28T15:43:57.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:57.289+0000 7f9d70296700 1 -- 172.21.15.23:0/2513489772 wait complete. 2024-03-28T15:43:57.606 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:57 smithi023 ceph-mon[32448]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:43:57.606 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:57.606 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:43:57.845 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T15:43:58.087 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:43:59.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:58 smithi023 ceph-mon[32448]: Added host smithi117 2024-03-28T15:43:59.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:59.079 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:43:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/4232477290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 msgr2=0x7f8d78106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/4232477290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d78106e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8d60009910 tx=0x7f8d60009c20 comp rx=0 tx=0).stop 2024-03-28T15:43:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/4232477290 shutdown_connections 2024-03-28T15:43:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/4232477290 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d78106e80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:59.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/4232477290 >> 172.21.15.23:0/4232477290 conn(0x7f8d780fc5d0 msgr2=0x7f8d780fe9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/4232477290 shutdown_connections 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/4232477290 wait complete. 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.609+0000 7f8d7e5f7700 1 Processor -- start 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d7e5f7700 1 -- start start 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d7e5f7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d7e5f7700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7810b0e0 con 0x7f8d78106aa0 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d77fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d77fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53636/0 (socket says 172.21.15.23:53636) 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d77fff700 1 -- 172.21.15.23:0/2275326039 learned_addr learned my addr 172.21.15.23:0/2275326039 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:43:59.612 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d77fff700 1 -- 172.21.15.23:0/2275326039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78196b00 con 0x7f8d78106aa0 2024-03-28T15:43:59.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.610+0000 7f8d77fff700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d60005850 tx=0x7f8d60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.611+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d60026070 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.611+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d6000bbd0 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.611+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8d60003970 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.611+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d78196df0 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.611+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d7819ae90 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f8d60029070 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d781021e0 con 0x7f8d78106aa0 2024-03-28T15:43:59.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d757fa700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 0x7f8d6403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:43:59.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8d6001e070 con 0x7f8d78106aa0 2024-03-28T15:43:59.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d777fe700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 0x7f8d6403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:43:59.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.612+0000 7f8d777fe700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 0x7f8d6403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d68006fd0 tx=0x7f8d68006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:43:59.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.617+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8d60013070 con 0x7f8d78106aa0 2024-03-28T15:43:59.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.830+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8d78197920 con 0x7f8d64038fc0 2024-03-28T15:43:59.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.831+0000 7f8d757fa700 1 -- 172.21.15.23:0/2275326039 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f8d78197920 con 0x7f8d64038fc0 2024-03-28T15:43:59.832 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:43:59.832 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}] 2024-03-28T15:43:59.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 msgr2=0x7f8d6403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:59.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 0x7f8d6403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8d68006fd0 tx=0x7f8d68006e40 comp rx=0 tx=0).stop 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 msgr2=0x7f8d781965c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8d60005850 tx=0x7f8d60005030 comp rx=0 tx=0).stop 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 shutdown_connections 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8d64038fc0 0x7f8d6403b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 --2- 172.21.15.23:0/2275326039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8d78106aa0 0x7f8d781965c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 >> 172.21.15.23:0/2275326039 conn(0x7f8d780fc5d0 msgr2=0x7f8d781085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 shutdown_connections 2024-03-28T15:43:59.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:43:59.833+0000 7f8d7e5f7700 1 -- 172.21.15.23:0/2275326039 wait complete. 2024-03-28T15:44:00.134 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:43:59 smithi023 ceph-mon[32448]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:00.481 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi120 2024-03-28T15:44:00.481 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:44:00.481 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.conf 2024-03-28T15:44:00.500 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:44:00.500 DEBUG:teuthology.orchestra.run.smithi120:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:00.557 INFO:tasks.cephadm:Adding host smithi120 to orchestrator... 2024-03-28T15:44:00.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch host add smithi120 2024-03-28T15:44:00.798 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:01.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:00 smithi023 ceph-mon[32448]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:44:01.163 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:02.085 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:01 smithi023 ceph-mon[32448]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 -- 172.21.15.23:0/3262657863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 msgr2=0x7f1968100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 --2- 172.21.15.23:0/3262657863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968100830 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1958009910 tx=0x7f1958009c20 comp rx=0 tx=0).stop 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 -- 172.21.15.23:0/3262657863 shutdown_connections 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 --2- 172.21.15.23:0/3262657863 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968100830 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 -- 172.21.15.23:0/3262657863 >> 172.21.15.23:0/3262657863 conn(0x7f19680fbce0 msgr2=0x7f19680fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 -- 172.21.15.23:0/3262657863 shutdown_connections 2024-03-28T15:44:02.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.284+0000 7f196cf07700 1 -- 172.21.15.23:0/3262657863 wait complete. 2024-03-28T15:44:02.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.285+0000 7f196cf07700 1 Processor -- start 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.285+0000 7f196cf07700 1 -- start start 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.285+0000 7f196cf07700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.285+0000 7f196cf07700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196810a820 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:53644/0 (socket says 172.21.15.23:53644) 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196659c700 1 -- 172.21.15.23:0/1063082418 learned_addr learned my addr 172.21.15.23:0/1063082418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196659c700 1 -- 172.21.15.23:0/1063082418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968107f70 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196659c700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1958005850 tx=0x7f1958005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1958026070 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f195800bc30 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1968106620 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1958003a20 con 0x7f1968100450 2024-03-28T15:44:02.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.286+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1968106ae0 con 0x7f1968100450 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.287+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1958029070 con 0x7f1968100450 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.287+0000 7f19577fe700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 0x7f195003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.287+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f195801e070 con 0x7f1968100450 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.287+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1968101930 con 0x7f1968100450 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.287+0000 7f1965d9b700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 0x7f195003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:02.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.288+0000 7f1965d9b700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 0x7f195003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f195c006fd0 tx=0x7f195c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:02.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.292+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1958030070 con 0x7f1968100450 2024-03-28T15:44:02.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:02.503+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}) v1 -- 0x7f19680637c0 con 0x7f1950038fc0 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi120", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:04.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:44:04.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:04.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:04.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:44:04.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: Deploying cephadm binary to smithi120 2024-03-28T15:44:04.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:03 smithi023 ceph-mon[32448]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:05.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:04 smithi023 ceph-mon[32448]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:05.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:04 smithi023 ceph-mon[32448]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:44:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:05 smithi023 ceph-mon[32448]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:06.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.727+0000 7f19577fe700 1 -- 172.21.15.23:0/1063082418 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f19680637c0 con 0x7f1950038fc0 2024-03-28T15:44:06.729 INFO:teuthology.orchestra.run.smithi023.stdout:Added host 'smithi120' with addr '172.21.15.120' 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 msgr2=0x7f195003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 0x7f195003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f195c006fd0 tx=0x7f195c006e40 comp rx=0 tx=0).stop 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 msgr2=0x7f1968106080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1958005850 tx=0x7f1958005030 comp rx=0 tx=0).stop 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 shutdown_connections 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1950038fc0 0x7f195003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 --2- 172.21.15.23:0/1063082418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1968100450 0x7f1968106080 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:06.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 >> 172.21.15.23:0/1063082418 conn(0x7f19680fbce0 msgr2=0x7f1968108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 shutdown_connections 2024-03-28T15:44:06.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:06.730+0000 7f196cf07700 1 -- 172.21.15.23:0/1063082418 wait complete. 2024-03-28T15:44:07.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:07.297 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:07.298 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:07.298 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:07.298 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:07.327 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-28T15:44:07.568 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:08.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:08 smithi023 ceph-mon[32448]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:08.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:08 smithi023 ceph-mon[32448]: Added host smithi120 2024-03-28T15:44:08.328 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:08 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:08.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:08 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/4178145872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 msgr2=0x7fb2c81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:09.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/4178145872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c81021c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb2b800b0a0 tx=0x7fb2b800b3b0 comp rx=0 tx=0).stop 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/4178145872 shutdown_connections 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/4178145872 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c81021c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/4178145872 >> 172.21.15.23:0/4178145872 conn(0x7fb2c80fd6d0 msgr2=0x7fb2c80ffaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/4178145872 shutdown_connections 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.075+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/4178145872 wait complete. 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cfdb0700 1 Processor -- start 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cfdb0700 1 -- start start 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cfdb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cfdb0700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2c810c1b0 con 0x7fb2c8101de0 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cdb4c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cdb4c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37244/0 (socket says 172.21.15.23:37244) 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.076+0000 7fb2cdb4c700 1 -- 172.21.15.23:0/3652274631 learned_addr learned my addr 172.21.15.23:0/3652274631 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:44:09.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2cdb4c700 1 -- 172.21.15.23:0/3652274631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c8076380 con 0x7fb2c8101de0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2cdb4c700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8015040 tx=0x7fb2b8003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b8017040 con 0x7fb2c8101de0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2b8022070 con 0x7fb2c8101de0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2c8074a30 con 0x7fb2c8101de0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2b801d5d0 con 0x7fb2c8101de0 2024-03-28T15:44:09.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.077+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2c8074ef0 con 0x7fb2c8101de0 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.078+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb2b8004000 con 0x7fb2c8101de0 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.078+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2c81032c0 con 0x7fb2c8101de0 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.078+0000 7fb2c6ffd700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 0x7fb2b003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.078+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb2b800f050 con 0x7fb2c8101de0 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.078+0000 7fb2cd34b700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 0x7fb2b003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:09.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.079+0000 7fb2cd34b700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 0x7fb2b003b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006fd0 tx=0x7fb2bc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:09.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.083+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb2b8007e20 con 0x7fb2c8101de0 2024-03-28T15:44:09.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.295+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2c80759c0 con 0x7fb2b0038fc0 2024-03-28T15:44:09.297 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.296+0000 7fb2c6ffd700 1 -- 172.21.15.23:0/3652274631 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fb2c80759c0 con 0x7fb2b0038fc0 2024-03-28T15:44:09.297 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:44:09.297 INFO:teuthology.orchestra.run.smithi023.stdout:[{"addr": "172.21.15.23", "hostname": "smithi023", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}, {"addr": "172.21.15.120", "hostname": "smithi120", "labels": [], "status": ""}] 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 msgr2=0x7fb2b003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 0x7fb2b003b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc006fd0 tx=0x7fb2bc006e40 comp rx=0 tx=0).stop 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 msgr2=0x7fb2c8074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8015040 tx=0x7fb2b8003d50 comp rx=0 tx=0).stop 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 shutdown_connections 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb2b0038fc0 0x7fb2b003b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:09.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 --2- 172.21.15.23:0/3652274631 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2c8101de0 0x7fb2c8074490 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:09.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 >> 172.21.15.23:0/3652274631 conn(0x7fb2c80fd6d0 msgr2=0x7fb2c81037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:09.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 shutdown_connections 2024-03-28T15:44:09.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:09.298+0000 7fb2cfdb0700 1 -- 172.21.15.23:0/3652274631 wait complete. 2024-03-28T15:44:09.886 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:09 smithi023 ceph-mon[32448]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:09.886 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:09.915 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-28T15:44:09.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-28T15:44:10.154 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:10.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:10 smithi023 ceph-mon[32448]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 -- 172.21.15.23:0/1638380923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 msgr2=0x7ff20c1021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 --2- 172.21.15.23:0/1638380923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c1021b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009910 tx=0x7ff1fc009c20 comp rx=0 tx=0).stop 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 -- 172.21.15.23:0/1638380923 shutdown_connections 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 --2- 172.21.15.23:0/1638380923 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c1021b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 -- 172.21.15.23:0/1638380923 >> 172.21.15.23:0/1638380923 conn(0x7ff20c0fd6a0 msgr2=0x7ff20c0ffac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:11.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 -- 172.21.15.23:0/1638380923 shutdown_connections 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.671+0000 7ff210e9e700 1 -- 172.21.15.23:0/1638380923 wait complete. 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff210e9e700 1 Processor -- start 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff210e9e700 1 -- start start 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff210e9e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff210e9e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c10c1a0 con 0x7ff20c101dd0 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff20a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff20a59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37268/0 (socket says 172.21.15.23:37268) 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.672+0000 7ff20a59c700 1 -- 172.21.15.23:0/2824092029 learned_addr learned my addr 172.21.15.23:0/2824092029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff20a59c700 1 -- 172.21.15.23:0/2824092029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c197c80 con 0x7ff20c101dd0 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff20a59c700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc006010 tx=0x7ff1fc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1fc026070 con 0x7ff20c101dd0 2024-03-28T15:44:11.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1fc00bbd0 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c197f70 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1fc003910 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.673+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c19c010 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.674+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff1fc029070 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.674+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c1032b0 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.674+0000 7ff1fb7fe700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 0x7ff1f403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.674+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff1fc01e070 con 0x7ff20c101dd0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.674+0000 7ff209d9b700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 0x7ff1f403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:11.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.675+0000 7ff209d9b700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 0x7ff1f403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff200006fd0 tx=0x7ff200006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:11.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.679+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1fc013070 con 0x7ff20c101dd0 2024-03-28T15:44:11.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:11.892+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff20c198aa0 con 0x7ff20c101dd0 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi120", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:12.769 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.768+0000 7ff1fb7fe700 1 -- 172.21.15.23:0/2824092029 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff1fc030090 con 0x7ff20c101dd0 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.770+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 msgr2=0x7ff1f403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.770+0000 7ff210e9e700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 0x7ff1f403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff200006fd0 tx=0x7ff200006e40 comp rx=0 tx=0).stop 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.770+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 msgr2=0x7ff20c197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.770+0000 7ff210e9e700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc006010 tx=0x7ff1fc005030 comp rx=0 tx=0).stop 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 shutdown_connections 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1f4039020 0x7ff1f403b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 --2- 172.21.15.23:0/2824092029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff20c101dd0 0x7ff20c197740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 >> 172.21.15.23:0/2824092029 conn(0x7ff20c0fd6a0 msgr2=0x7ff20c104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 shutdown_connections 2024-03-28T15:44:12.772 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:12.771+0000 7ff210e9e700 1 -- 172.21.15.23:0/2824092029 wait complete. 2024-03-28T15:44:12.773 INFO:teuthology.orchestra.run.smithi023.stderr:adjusted tunables profile to default 2024-03-28T15:44:13.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:44:13.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2824092029' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-28T15:44:13.064 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:13.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:13.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:13.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:13.065 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:13.350 INFO:tasks.cephadm:Adding mon.a on smithi023 2024-03-28T15:44:13.350 INFO:tasks.cephadm:Adding mon.b on smithi117 2024-03-28T15:44:13.350 INFO:tasks.cephadm:Adding mon.c on smithi120 2024-03-28T15:44:13.351 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi023:172.21.15.23=a;smithi117:172.21.15.117=b;smithi120:172.21.15.120=c' 2024-03-28T15:44:13.579 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:13 smithi023 ceph-mon[32448]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:13 smithi023 ceph-mon[32448]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:44:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:13 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2824092029' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-28T15:44:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:13 smithi023 ceph-mon[32448]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 -- 172.21.15.120:0/3673042591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 msgr2=0x7fb3d01021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 --2- 172.21.15.120:0/3673042591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01021a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc00b0a0 tx=0x7fb3cc00b3b0 comp rx=0 tx=0).stop 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 -- 172.21.15.120:0/3673042591 shutdown_connections 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 --2- 172.21.15.120:0/3673042591 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01021a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 -- 172.21.15.120:0/3673042591 >> 172.21.15.120:0/3673042591 conn(0x7fb3d00fd650 msgr2=0x7fb3d00ffa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.889+0000 7fb3d8214700 1 -- 172.21.15.120:0/3673042591 shutdown_connections 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d8214700 1 -- 172.21.15.120:0/3673042591 wait complete. 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d8214700 1 Processor -- start 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d8214700 1 -- start start 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d8214700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d8214700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d010c190 con 0x7fb3d0101dc0 2024-03-28T15:44:14.892 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d5fb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d5fb0700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.120:52010/0 (socket says 172.21.15.120:52010) 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.890+0000 7fb3d5fb0700 1 -- 172.21.15.120:0/2671183097 learned_addr learned my addr 172.21.15.120:0/2671183097 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3d5fb0700 1 -- 172.21.15.120:0/2671183097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d0197b30 con 0x7fb3d0101dc0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3d5fb0700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc015040 tx=0x7fb3cc003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3cc017040 con 0x7fb3d0101dc0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d0197dc0 con 0x7fb3d0101dc0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc022070 con 0x7fb3d0101dc0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3cc01d5d0 con 0x7fb3d0101dc0 2024-03-28T15:44:14.893 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.891+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d019bec0 con 0x7fb3d0101dc0 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.892+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d01032a0 con 0x7fb3d0101dc0 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.892+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb3cc004000 con 0x7fb3d0101dc0 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.892+0000 7fb3c6ffd700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 0x7fb3bc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.892+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3cc00f050 con 0x7fb3d0101dc0 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.893+0000 7fb3d57af700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 0x7fb3bc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:14.894 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.893+0000 7fb3d57af700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 0x7fb3bc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0006fd0 tx=0x7fb3c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:14.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:14.898+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3cc007e20 con 0x7fb3d0101dc0 2024-03-28T15:44:15.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.102+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi117:172.21.15.117=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3d0198950 con 0x7fb3bc038fc0 2024-03-28T15:44:15.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.110+0000 7fb3c6ffd700 1 -- 172.21.15.120:0/2671183097 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb3d0198950 con 0x7fb3bc038fc0 2024-03-28T15:44:15.111 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mon update... 2024-03-28T15:44:15.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 msgr2=0x7fb3bc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:15.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 0x7fb3bc03b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb3c0006fd0 tx=0x7fb3c0006e40 comp rx=0 tx=0).stop 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 msgr2=0x7fb3d01975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc015040 tx=0x7fb3cc003d50 comp rx=0 tx=0).stop 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 shutdown_connections 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb3bc038fc0 0x7fb3bc03b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.112+0000 7fb3d8214700 1 --2- 172.21.15.120:0/2671183097 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb3d0101dc0 0x7fb3d01975f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.113+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 >> 172.21.15.120:0/2671183097 conn(0x7fb3d00fd650 msgr2=0x7fb3d0103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.113+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 shutdown_connections 2024-03-28T15:44:15.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:15.113+0000 7fb3d8214700 1 -- 172.21.15.120:0/2671183097 wait complete. 2024-03-28T15:44:15.737 DEBUG:teuthology.orchestra.run.smithi117:mon.b> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.b.service 2024-03-28T15:44:15.740 DEBUG:teuthology.orchestra.run.smithi120:mon.c> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.c.service 2024-03-28T15:44:15.742 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:44:15.743 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:44:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi023:172.21.15.23=a;smithi117:172.21.15.117=b;smithi120:172.21.15.120=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: Saving service mon spec with placement smithi023:172.21.15.23=a;smithi117:172.21.15.117=b;smithi120:172.21.15.120=c;count:3 2024-03-28T15:44:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:16 smithi023 ceph-mon[32448]: Deploying daemon mon.c on smithi120 2024-03-28T15:44:16.537 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:18 smithi023 ceph-mon[32448]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 -- 172.21.15.120:0/1180456152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 msgr2=0x7f1570106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 --2- 172.21.15.120:0/1180456152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f1570106e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1558009910 tx=0x7f1558009c20 comp rx=0 tx=0).stop 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 -- 172.21.15.120:0/1180456152 shutdown_connections 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 --2- 172.21.15.120:0/1180456152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f1570106e50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 -- 172.21.15.120:0/1180456152 >> 172.21.15.120:0/1180456152 conn(0x7f15700fc580 msgr2=0x7f15700fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 -- 172.21.15.120:0/1180456152 shutdown_connections 2024-03-28T15:44:18.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.437+0000 7f157496c700 1 -- 172.21.15.120:0/1180456152 wait complete. 2024-03-28T15:44:18.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f157496c700 1 Processor -- start 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f157496c700 1 -- start start 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f157496c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f157496c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157010b0b0 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f156e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f156e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.120:52016/0 (socket says 172.21.15.120:52016) 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.438+0000 7f156e59c700 1 -- 172.21.15.120:0/1047403487 learned_addr learned my addr 172.21.15.120:0/1047403487 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f156e59c700 1 -- 172.21.15.120:0/1047403487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570196af0 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f156e59c700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1558005850 tx=0x7f1558005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1558026070 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1570196de0 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f155800bbd0 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1558003970 con 0x7f1570106a70 2024-03-28T15:44:18.440 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.439+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157019ae80 con 0x7f1570106a70 2024-03-28T15:44:18.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.440+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15701021b0 con 0x7f1570106a70 2024-03-28T15:44:18.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.440+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1558029070 con 0x7f1570106a70 2024-03-28T15:44:18.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.440+0000 7f15677fe700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 0x7f155c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:18.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.440+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f155801e070 con 0x7f1570106a70 2024-03-28T15:44:18.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.440+0000 7f156dd9b700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 0x7f155c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:18.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.441+0000 7f156dd9b700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 0x7f155c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1560006fd0 tx=0x7f1560006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:18.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.446+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1558013070 con 0x7f1570106a70 2024-03-28T15:44:18.732 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.732+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1570197910 con 0x7f1570106a70 2024-03-28T15:44:18.733 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.732+0000 7f15677fe700 1 -- 172.21.15.120:0/1047403487 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f1558030090 con 0x7f1570106a70 2024-03-28T15:44:18.733 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:44:18.733 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":1,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","modified":"2024-03-28T15:42:24.258212Z","created":"2024-03-28T15:42:24.258212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-28T15:44:18.735 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 msgr2=0x7f155c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 0x7f155c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1560006fd0 tx=0x7f1560006e40 comp rx=0 tx=0).stop 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 msgr2=0x7f15701965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1558005850 tx=0x7f1558005030 comp rx=0 tx=0).stop 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 shutdown_connections 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f155c038fc0 0x7f155c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 --2- 172.21.15.120:0/1047403487 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1570106a70 0x7f15701965b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 >> 172.21.15.120:0/1047403487 conn(0x7f15700fc580 msgr2=0x7f1570108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 shutdown_connections 2024-03-28T15:44:18.736 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:18.735+0000 7f157496c700 1 -- 172.21.15.120:0/1047403487 wait complete. 2024-03-28T15:44:18.737 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 1 2024-03-28T15:44:19.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:19 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1047403487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:20.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:20 smithi023 ceph-mon[32448]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:21.202 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:44:21.202 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:44:21.229 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:20 smithi120 systemd[1]: Started Ceph mon.c for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:44:21.435 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 -- 172.21.15.120:0/1761286799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb00057d0 msgr2=0x7efeb0007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 --2- 172.21.15.120:0/1761286799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb00057d0 0x7efeb0007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efec4100270 tx=0x7efebc00cec0 comp rx=0 tx=0).stop 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 -- 172.21.15.120:0/1761286799 shutdown_connections 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 --2- 172.21.15.120:0/1761286799 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efeb00057d0 0x7efeb0007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 -- 172.21.15.120:0/1761286799 >> 172.21.15.120:0/1761286799 conn(0x7efec4077280 msgr2=0x7efec4077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 -- 172.21.15.120:0/1761286799 shutdown_connections 2024-03-28T15:44:25.963 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.962+0000 7efecc48b700 1 -- 172.21.15.120:0/1761286799 wait complete. 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 Processor -- start 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 -- start start 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 0x7efec41998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4107af0 con 0x7efec419fc10 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.963+0000 7efecc48b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4107970 con 0x7efeb00057d0 2024-03-28T15:44:25.965 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:59420/0 (socket says 172.21.15.120:59420) 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 -- 172.21.15.120:0/2033702399 learned_addr learned my addr 172.21.15.120:0/2033702399 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efec9a26700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 0x7efec41998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 -- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 msgr2=0x7efec419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 -- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 msgr2=0x7efec419f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efeca227700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efec9a26700 1 -- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 msgr2=0x7efec419f6d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efec9a26700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efec9a26700 1 -- 172.21.15.120:0/2033702399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec41a0250 con 0x7efec419fc10 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.964+0000 7efec9a26700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 0x7efec41998f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b4d0 tx=0x7efeb400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:25.966 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.965+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7efeb4011630 con 0x7efec419fc10 2024-03-28T15:44:25.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.965+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efeb4011cb0 con 0x7efec419fc10 2024-03-28T15:44:25.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.965+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7efeb4010e60 con 0x7efec419fc10 2024-03-28T15:44:25.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.965+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec4199e90 con 0x7efec419fc10 2024-03-28T15:44:25.967 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.965+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec419a3f0 con 0x7efec419fc10 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.966+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7efeb4018020 con 0x7efec419fc10 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.966+0000 7efebb7fe700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 0x7efea803b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.966+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4101a30 con 0x7efec419fc10 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.966+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efeb404ba50 con 0x7efec419fc10 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.966+0000 7efeca227700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 0x7efea803b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:25.968 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.967+0000 7efeca227700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 0x7efea803b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efebc01a040 tx=0x7efebc005710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:25.972 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:25.971+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efeb401f020 con 0x7efec419fc10 2024-03-28T15:44:26.248 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.248+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efec419a760 con 0x7efec419fc10 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: mon.a calling monitor election 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.249 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: mon.c calling monitor election 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: fsmap 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: mgrmap e13: x(active, since 51s) 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: overall HEALTH_OK 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:26.250 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:26.251 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.249+0000 7efebb7fe700 1 -- 172.21.15.120:0/2033702399 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7efeb4018570 con 0x7efec419fc10 2024-03-28T15:44:26.251 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:44:26.251 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":2,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","modified":"2024-03-28T15:44:20.910234Z","created":"2024-03-28T15:42:24.258212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.250+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 msgr2=0x7efea803b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.250+0000 7efecc48b700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 0x7efea803b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efebc01a040 tx=0x7efebc005710 comp rx=0 tx=0).stop 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 msgr2=0x7efec41998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 0x7efec41998f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efeb400b4d0 tx=0x7efeb400b890 comp rx=0 tx=0).stop 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 shutdown_connections 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efea8038d60 0x7efea803b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efeb00057d0 0x7efec419f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 --2- 172.21.15.120:0/2033702399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efec419fc10 0x7efec41998f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:26.252 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 >> 172.21.15.120:0/2033702399 conn(0x7efec4077280 msgr2=0x7efec41032d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:26.253 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 shutdown_connections 2024-03-28T15:44:26.253 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:26.251+0000 7efecc48b700 1 -- 172.21.15.120:0/2033702399 wait complete. 2024-03-28T15:44:26.253 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 2 2024-03-28T15:44:26.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: mon.a calling monitor election 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: mon.c calling monitor election 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: monmap e2: 2 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],c=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:44:26.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: fsmap 2024-03-28T15:44:26.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:26.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: mgrmap e13: x(active, since 51s) 2024-03-28T15:44:26.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: overall HEALTH_OK 2024-03-28T15:44:26.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:26.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:27.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:27.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:27.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:27.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: Deploying daemon mon.b on smithi117 2024-03-28T15:44:27.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/2033702399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:27.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:27.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:27.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:27.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:27.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: Deploying daemon mon.b on smithi117 2024-03-28T15:44:27.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/2033702399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:27.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:27.875 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-28T15:44:27.876 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-28T15:44:28.027 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:44:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:44:27.909+0000 7f28637d6700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-28T15:44:28.119 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:44:28.232 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:28 smithi120 ceph-mon[36826]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:28.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:28 smithi023 ceph-mon[32448]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:29.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.437+0000 7f7fbba77700 1 -- 172.21.15.120:0/3708331664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4100dc0 msgr2=0x7f7fb41011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:29.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.437+0000 7f7fbba77700 1 --2- 172.21.15.120:0/3708331664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4100dc0 0x7f7fb41011a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7fac00b0a0 tx=0x7f7fac00b3b0 comp rx=0 tx=0).stop 2024-03-28T15:44:29.438 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.437+0000 7f7fbba77700 1 -- 172.21.15.120:0/3708331664 shutdown_connections 2024-03-28T15:44:29.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.437+0000 7f7fbba77700 1 --2- 172.21.15.120:0/3708331664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4100dc0 0x7f7fb41011a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:29.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.437+0000 7f7fbba77700 1 -- 172.21.15.120:0/3708331664 >> 172.21.15.120:0/3708331664 conn(0x7f7fb40fc650 msgr2=0x7f7fb40fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:29.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.438+0000 7f7fbba77700 1 -- 172.21.15.120:0/3708331664 shutdown_connections 2024-03-28T15:44:29.439 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.438+0000 7f7fbba77700 1 -- 172.21.15.120:0/3708331664 wait complete. 2024-03-28T15:44:29.441 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.438+0000 7f7fbba77700 1 Processor -- start 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.438+0000 7f7fbba77700 1 -- start start 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fbba77700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fbba77700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 0x7f7fb4199120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fbba77700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb410b190 con 0x7f7fb4198ca0 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fbba77700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb410b010 con 0x7f7fb4100dc0 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44316/0 (socket says 172.21.15.120:44316) 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 -- 172.21.15.120:0/1946229203 learned_addr learned my addr 172.21.15.120:0/1946229203 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9012700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 0x7f7fb4199120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 -- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 msgr2=0x7f7fb4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 -- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 msgr2=0x7f7fb4198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-28T15:44:29.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9813700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9012700 1 -- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 msgr2=0x7f7fb4198760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9012700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9012700 1 -- 172.21.15.120:0/1946229203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb41a17d0 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fb9012700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 0x7f7fb4199120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b4d0 tx=0x7f7fa400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7fa4011630 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fa4011cb0 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb41a19a0 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.439+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f7fa4010e60 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.440+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb41a1f30 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.440+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f7fa4018020 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.440+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb41022a0 con 0x7f7fb4198ca0 2024-03-28T15:44:29.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.441+0000 7f7faaffd700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 0x7f7fa003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:29.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.441+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7fa404ba00 con 0x7f7fb4198ca0 2024-03-28T15:44:29.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.441+0000 7f7fb9813700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 0x7f7fa003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:29.444 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.441+0000 7f7fb9813700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 0x7f7fa003b1c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009e60 tx=0x7f7fac006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:29.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.446+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7fa401f020 con 0x7f7fb4198ca0 2024-03-28T15:44:29.651 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.651+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa4010510 con 0x7f7fb4198ca0 2024-03-28T15:44:29.723 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:29.722+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fb41a2250 con 0x7f7fb4198ca0 2024-03-28T15:44:29.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:29 smithi117 systemd[1]: Started Ceph mon.b for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:44:34.700 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.699+0000 7f7faaffd700 1 -- 172.21.15.120:0/1946229203 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1285 (secure 0 0 0) 0x7f7fa4014170 con 0x7f7fb4198ca0 2024-03-28T15:44:34.700 INFO:teuthology.orchestra.run.smithi120.stdout: 2024-03-28T15:44:34.700 INFO:teuthology.orchestra.run.smithi120.stdout:{"epoch":3,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","modified":"2024-03-28T15:44:29.648745Z","created":"2024-03-28T15:42:24.258212Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:3300","nonce":0},{"type":"v1","addr":"172.21.15.23:6789","nonce":0}]},"addr":"172.21.15.23:6789/0","public_addr":"172.21.15.23:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:3300","nonce":0},{"type":"v1","addr":"172.21.15.120:6789","nonce":0}]},"addr":"172.21.15.120:6789/0","public_addr":"172.21.15.120:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 msgr2=0x7f7fa003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 0x7f7fa003b1c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7fac009e60 tx=0x7f7fac006230 comp rx=0 tx=0).stop 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 msgr2=0x7f7fb4199120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 0x7f7fb4199120 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fa400b4d0 tx=0x7f7fa400b890 comp rx=0 tx=0).stop 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 shutdown_connections 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7fa0038d00 0x7f7fa003b1c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7fb4100dc0 0x7f7fb4198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 --2- 172.21.15.120:0/1946229203 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7fb4198ca0 0x7f7fb4199120 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 >> 172.21.15.120:0/1946229203 conn(0x7f7fb40fc650 msgr2=0x7f7fb4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 shutdown_connections 2024-03-28T15:44:34.703 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:34.702+0000 7f7fbba77700 1 -- 172.21.15.120:0/1946229203 wait complete. 2024-03-28T15:44:34.704 INFO:teuthology.orchestra.run.smithi120.stderr:dumped monmap epoch 3 2024-03-28T15:44:35.034 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: mon.a calling monitor election 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: mon.c calling monitor election 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/1946229203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: mon.b calling monitor election 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.035 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0],c=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: fsmap 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: mgrmap e13: x(active, since 60s) 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: overall HEALTH_OK 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.036 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:44:35.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: mon.a calling monitor election 2024-03-28T15:44:35.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: mon.c calling monitor election 2024-03-28T15:44:35.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/1946229203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: mon.b calling monitor election 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0],c=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: fsmap 2024-03-28T15:44:35.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:35.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: mgrmap e13: x(active, since 60s) 2024-03-28T15:44:35.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: overall HEALTH_OK 2024-03-28T15:44:35.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:44:35.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: mon.a calling monitor election 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: mon.c calling monitor election 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1946229203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: mon.b calling monitor election 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-28T15:44:35.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: monmap e3: 3 mons at {a=[v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0],b=[v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0],c=[v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: fsmap 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: osdmap e4: 0 total, 0 up, 0 in 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: mgrmap e13: x(active, since 60s) 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: overall HEALTH_OK 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.172 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.328 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-28T15:44:35.328 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-28T15:44:35.578 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:35.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:35 smithi023 ceph-mon[32448]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:35.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:35.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:35.991 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:36.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:35 smithi117 ceph-mon[36002]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:36.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:36.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:36.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:36.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:36.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:35 smithi120 ceph-mon[36826]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:36.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:36.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:36.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:36.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:44:37.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 -- 172.21.15.23:0/266794312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6978101dd0 msgr2=0x7f69781021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 --2- 172.21.15.23:0/266794312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6978101dd0 0x7f69781021b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f696c009910 tx=0x7f696c009c20 comp rx=0 tx=0).stop 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 -- 172.21.15.23:0/266794312 shutdown_connections 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 --2- 172.21.15.23:0/266794312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6978101dd0 0x7f69781021b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 -- 172.21.15.23:0/266794312 >> 172.21.15.23:0/266794312 conn(0x7f69780fd680 msgr2=0x7f69780ffaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 -- 172.21.15.23:0/266794312 shutdown_connections 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.228+0000 7f697f040700 1 -- 172.21.15.23:0/266794312 wait complete. 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 Processor -- start 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 -- start start 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 0x7f6978074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69780749d0 0x7f6978074e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:37.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697810c1a0 con 0x7f69781a3ae0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697810c020 con 0x7f69780749d0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697f040700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697810c320 con 0x7f6978101dd0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697d5dd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697d5dd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:35070/0 (socket says 172.21.15.23:35070) 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.229+0000 7f697d5dd700 1 -- 172.21.15.23:0/597245964 learned_addr learned my addr 172.21.15.23:0/597245964 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f6977fff700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69780749d0 0x7f6978074e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697cddc700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 0x7f6978074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 -- 172.21.15.23:0/597245964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 msgr2=0x7f6978074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 0x7f6978074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 -- 172.21.15.23:0/597245964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69780749d0 msgr2=0x7f6978074e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69780749d0 0x7f6978074e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 -- 172.21.15.23:0/597245964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69781a6600 con 0x7f69781a3ae0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697cddc700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 0x7f6978074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697d5dd700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f697000ba50 tx=0x7f697000be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970019070 con 0x7f69781a3ae0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6970014440 con 0x7f69781a3ae0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69781a68f0 con 0x7f69781a3ae0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970007590 con 0x7f69781a3ae0 2024-03-28T15:44:37.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.230+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69781a6de0 con 0x7f69781a3ae0 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.231+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f697001c070 con 0x7f69781a3ae0 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.231+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69780637c0 con 0x7f69781a3ae0 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.231+0000 7f6975ffb700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 0x7f696803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.231+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f697004b770 con 0x7f69781a3ae0 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.232+0000 7f697cddc700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 0x7f696803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:37.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.232+0000 7f697cddc700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 0x7f696803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f696c00b5c0 tx=0x7f696c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:37.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.236+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6970014910 con 0x7f69781a3ae0 2024-03-28T15:44:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.341 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.341 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.341 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:37.341 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:37.341 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:37.392 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.392 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.392 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.392 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:37.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:37.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.447+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f69781a7170 con 0x7f69781a3ae0 2024-03-28T15:44:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.447+0000 7f6975ffb700 1 -- 172.21.15.23:0/597245964 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f69700145f0 con 0x7f69781a3ae0 2024-03-28T15:44:37.449 INFO:teuthology.orchestra.run.smithi023.stdout:# minimal ceph.conf for a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:44:37.449 INFO:teuthology.orchestra.run.smithi023.stdout:[global] 2024-03-28T15:44:37.449 INFO:teuthology.orchestra.run.smithi023.stdout: fsid = a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:44:37.449 INFO:teuthology.orchestra.run.smithi023.stdout: mon_host = [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 msgr2=0x7f696803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 0x7f696803b910 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f696c00b5c0 tx=0x7f696c0058e0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 msgr2=0x7f69781a5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f697000ba50 tx=0x7f697000be10 comp rx=0 tx=0).stop 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 shutdown_connections 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6968039450 0x7f696803b910 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69781a3ae0 0x7f69781a5ed0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6978101dd0 0x7f6978074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 --2- 172.21.15.23:0/597245964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69780749d0 0x7f6978074e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 >> 172.21.15.23:0/597245964 conn(0x7f69780fd680 msgr2=0x7f69781037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 shutdown_connections 2024-03-28T15:44:37.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:37.450+0000 7f697f040700 1 -- 172.21.15.23:0/597245964 wait complete. 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:44:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:37.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:38.131 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-28T15:44:38.132 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:44:38.132 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:44:38.161 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:44:38.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:38.228 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:44:38.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:44:38.263 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:44:38.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:38.334 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:44:38.334 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.conf 2024-03-28T15:44:38.369 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:44:38.369 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: Reconfiguring daemon mon.a on smithi023 2024-03-28T15:44:38.416 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:38 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/597245964' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:44:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: Reconfiguring daemon mon.a on smithi023 2024-03-28T15:44:38.436 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:38 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/597245964' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:38.439 INFO:tasks.cephadm:Adding mgr.x on smithi023 2024-03-28T15:44:38.439 INFO:tasks.cephadm:Adding mgr.y on smithi117 2024-03-28T15:44:38.439 INFO:tasks.cephadm:Adding mgr.z on smithi120 2024-03-28T15:44:38.439 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi023=x;smithi117=y;smithi120=z' 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: Reconfiguring mon.a (unknown last config time)... 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: Reconfiguring daemon mon.a on smithi023 2024-03-28T15:44:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:38 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/597245964' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:38.709 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:40.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:40.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: Reconfiguring daemon mon.b on smithi117 2024-03-28T15:44:40.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 -- 172.21.15.120:0/1255257399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41701033e0 msgr2=0x7f4170103800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 --2- 172.21.15.120:0/1255257399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41701033e0 0x7f4170103800 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4164009910 tx=0x7f4164009c20 comp rx=0 tx=0).stop 2024-03-28T15:44:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 -- 172.21.15.120:0/1255257399 shutdown_connections 2024-03-28T15:44:40.191 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 --2- 172.21.15.120:0/1255257399 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f41701033e0 0x7f4170103800 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 -- 172.21.15.120:0/1255257399 >> 172.21.15.120:0/1255257399 conn(0x7f41700fd280 msgr2=0x7f41700ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 -- 172.21.15.120:0/1255257399 shutdown_connections 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.188+0000 7f41751fb700 1 -- 172.21.15.120:0/1255257399 wait complete. 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.189+0000 7f41751fb700 1 Processor -- start 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.189+0000 7f41751fb700 1 -- start start 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 0x7f41701975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 0x7f417019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 0x7f417019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4170107e50 con 0x7f4170197fb0 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4170107cd0 con 0x7f4170197b30 2024-03-28T15:44:40.192 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f41751fb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4170107fd0 con 0x7f41701033e0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416ed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 0x7f41701975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 0x7f417019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 0x7f417019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416ed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 0x7f41701975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:45712/0 (socket says 172.21.15.120:45712) 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416ed9d700 1 -- 172.21.15.120:0/3964202028 learned_addr learned my addr 172.21.15.120:0/3964202028 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 0x7f417019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.120:57548/0 (socket says 172.21.15.120:57548) 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 -- 172.21.15.120:0/3964202028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 msgr2=0x7f41701975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 0x7f41701975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 -- 172.21.15.120:0/3964202028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 msgr2=0x7f417019cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 0x7f417019cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 -- 172.21.15.120:0/3964202028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417019fa00 con 0x7f4170197fb0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.190+0000 7f416f59e700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 0x7f417019f130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f416800bd50 tx=0x7f416800bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168019070 con 0x7f4170197fb0 2024-03-28T15:44:40.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f416800cbb0 con 0x7f4170197fb0 2024-03-28T15:44:40.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417019fcf0 con 0x7f4170197fb0 2024-03-28T15:44:40.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4168012620 con 0x7f4170197fb0 2024-03-28T15:44:40.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41701a0200 con 0x7f4170197fb0 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41700637c0 con 0x7f4170197fb0 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.191+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f416801c070 con 0x7f4170197fb0 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.192+0000 7f4157fff700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 0x7f415803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.192+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f416804b920 con 0x7f4170197fb0 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.192+0000 7f416ed9d700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 0x7f415803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:40.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.193+0000 7f416ed9d700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 0x7f415803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4164005200 tx=0x7f416401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:40.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.197+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f416800cdd0 con 0x7f4170197fb0 2024-03-28T15:44:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:44:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:40.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:40.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:40.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: Reconfiguring daemon mon.b on smithi117 2024-03-28T15:44:40.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.406 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.405+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi117=y;smithi120=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f41700625d0 con 0x7f4158039450 2024-03-28T15:44:40.412 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.412+0000 7f4157fff700 1 -- 172.21.15.120:0/3964202028 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f41700625d0 con 0x7f4158039450 2024-03-28T15:44:40.412 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mgr update... 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 msgr2=0x7f415803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 0x7f415803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4164005200 tx=0x7f416401a040 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 msgr2=0x7f417019f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 0x7f417019f130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f416800bd50 tx=0x7f416800bd80 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 shutdown_connections 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4158039450 0x7f415803b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4170197fb0 0x7f417019f130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41701033e0 0x7f41701975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 --2- 172.21.15.120:0/3964202028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4170197b30 0x7f417019cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 >> 172.21.15.120:0/3964202028 conn(0x7f41700fd280 msgr2=0x7f41700fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 shutdown_connections 2024-03-28T15:44:40.415 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:44:40.414+0000 7f41751fb700 1 -- 172.21.15.120:0/3964202028 wait complete. 2024-03-28T15:44:40.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:40.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: Reconfiguring mon.b (monmap changed)... 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: Reconfiguring daemon mon.b on smithi117 2024-03-28T15:44:40.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:40.935 DEBUG:teuthology.orchestra.run.smithi117:mgr.y> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.y.service 2024-03-28T15:44:40.941 DEBUG:teuthology.orchestra.run.smithi120:mgr.z> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.z.service 2024-03-28T15:44:40.944 INFO:tasks.cephadm:Deploying OSDs... 2024-03-28T15:44:40.944 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:44:40.944 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:44:40.963 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:44:40.964 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_1 2024-03-28T15:44:41.024 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 15:43:46.793081716 +0000 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 15:38:26.583305620 +0000 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 15:38:26.583305620 +0000 2024-03-28T15:44:41.025 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 15:38:26.583305620 +0000 2024-03-28T15:44:41.025 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:44:41.098 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T15:44:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T15:44:41.099 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000317218 s, 1.6 MB/s 2024-03-28T15:44:41.100 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:44:41.162 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_2 2024-03-28T15:44:41.221 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 15:43:46.829081403 +0000 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 15:38:26.923302865 +0000 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 15:38:26.923302865 +0000 2024-03-28T15:44:41.222 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 15:38:26.923302865 +0000 2024-03-28T15:44:41.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:44:41.290 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T15:44:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T15:44:41.291 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000306757 s, 1.7 MB/s 2024-03-28T15:44:41.292 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:44:41.349 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_3 2024-03-28T15:44:41.406 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 15:43:46.865081091 +0000 2024-03-28T15:44:41.407 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 15:38:27.265300093 +0000 2024-03-28T15:44:41.408 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 15:38:27.265300093 +0000 2024-03-28T15:44:41.408 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 15:38:27.265300093 +0000 2024-03-28T15:44:41.408 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:44:41.473 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T15:44:41.474 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T15:44:41.474 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000306247 s, 1.7 MB/s 2024-03-28T15:44:41.475 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:44:41.531 DEBUG:teuthology.orchestra.run.smithi023:> stat /dev/vg_nvme/lv_4 2024-03-28T15:44:41.587 INFO:teuthology.orchestra.run.smithi023.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:44:41.587 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.587 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-28T15:44:41.587 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 15:43:46.899080796 +0000 2024-03-28T15:44:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 15:38:27.604297345 +0000 2024-03-28T15:44:41.588 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 15:38:27.604297345 +0000 2024-03-28T15:44:41.588 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 15:38:27.604297345 +0000 2024-03-28T15:44:41.588 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:44:41.651 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi117=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: Saving service mgr spec with placement smithi023=x;smithi117=y;smithi120=z;count:3 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:41.652 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:41.653 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:41 smithi023 ceph-mon[32448]: Reconfiguring daemon mon.c on smithi120 2024-03-28T15:44:41.654 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records in 2024-03-28T15:44:41.654 INFO:teuthology.orchestra.run.smithi023.stderr:1+0 records out 2024-03-28T15:44:41.654 INFO:teuthology.orchestra.run.smithi023.stderr:512 bytes copied, 0.000261385 s, 2.0 MB/s 2024-03-28T15:44:41.655 DEBUG:teuthology.orchestra.run.smithi023:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:44:41.711 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-28T15:44:41.711 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:44:41.730 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:44:41.730 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-28T15:44:41.787 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:44:41.787 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.787 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-28 15:44:29.122965836 +0000 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-28 15:38:49.585027421 +0000 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-28 15:38:49.585027421 +0000 2024-03-28T15:44:41.788 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-28 15:38:49.585027421 +0000 2024-03-28T15:44:41.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi117=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: Saving service mgr spec with placement smithi023=x;smithi117=y;smithi120=z;count:3 2024-03-28T15:44:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:41.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:41.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:41.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:41 smithi117 ceph-mon[36002]: Reconfiguring daemon mon.c on smithi120 2024-03-28T15:44:41.825 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:41.825 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi023=x;smithi117=y;smithi120=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: Reconfiguring mon.c (monmap changed)... 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: Saving service mgr spec with placement smithi023=x;smithi117=y;smithi120=z;count:3 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-28T15:44:41.826 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-28T15:44:41.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:41.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:41.827 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:41 smithi120 ceph-mon[36826]: Reconfiguring daemon mon.c on smithi120 2024-03-28T15:44:41.850 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-28T15:44:41.850 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-28T15:44:41.850 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000287344 s, 1.8 MB/s 2024-03-28T15:44:41.851 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:44:41.909 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-28T15:44:41.966 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:44:41.966 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:41.966 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T15:44:41.966 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:41.967 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:41.967 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-28 15:44:29.122965836 +0000 2024-03-28T15:44:41.967 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-28 15:38:49.942020386 +0000 2024-03-28T15:44:41.967 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-28 15:38:49.942020386 +0000 2024-03-28T15:44:41.967 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-28 15:38:49.942020386 +0000 2024-03-28T15:44:41.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:44:42.035 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-28T15:44:42.035 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-28T15:44:42.035 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000229863 s, 2.2 MB/s 2024-03-28T15:44:42.036 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:44:42.094 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:42.150 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-28 15:44:29.123965816 +0000 2024-03-28T15:44:42.151 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-28 15:38:50.278013766 +0000 2024-03-28T15:44:42.151 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-28 15:38:50.278013766 +0000 2024-03-28T15:44:42.151 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-28 15:38:50.278013766 +0000 2024-03-28T15:44:42.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:44:42.218 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-28T15:44:42.218 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-28T15:44:42.218 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000200172 s, 2.6 MB/s 2024-03-28T15:44:42.219 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:44:42.275 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-28T15:44:42.331 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:44:42.331 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-28 15:44:29.123965816 +0000 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-28 15:38:50.619007046 +0000 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-28 15:38:50.619007046 +0000 2024-03-28T15:44:42.332 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-28 15:38:50.619007046 +0000 2024-03-28T15:44:42.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:44:42.398 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-28T15:44:42.398 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-28T15:44:42.398 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000213664 s, 2.4 MB/s 2024-03-28T15:44:42.399 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:44:42.456 DEBUG:teuthology.orchestra.run.smithi120:> set -ex 2024-03-28T15:44:42.456 DEBUG:teuthology.orchestra.run.smithi120:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T15:44:42.475 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T15:44:42.475 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_1 2024-03-28T15:44:42.532 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-28 15:44:20.394190387 +0000 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-28 15:38:46.346618490 +0000 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-28 15:38:46.346618490 +0000 2024-03-28T15:44:42.533 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-28 15:38:46.346618490 +0000 2024-03-28T15:44:42.533 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T15:44:42.601 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-28T15:44:42.602 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-28T15:44:42.602 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000277223 s, 1.8 MB/s 2024-03-28T15:44:42.603 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T15:44:42.661 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_2 2024-03-28T15:44:42.717 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-28 15:44:20.395190365 +0000 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-28 15:38:46.677610966 +0000 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-28 15:38:46.677610966 +0000 2024-03-28T15:44:42.718 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-28 15:38:46.677610966 +0000 2024-03-28T15:44:42.718 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T15:44:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-28T15:44:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-28T15:44:42.785 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000232289 s, 2.2 MB/s 2024-03-28T15:44:42.787 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T15:44:42.842 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_3 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-28 15:44:20.395190365 +0000 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-28 15:38:47.015603283 +0000 2024-03-28T15:44:42.898 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-28 15:38:47.015603283 +0000 2024-03-28T15:44:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-28 15:38:47.015603283 +0000 2024-03-28T15:44:42.899 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T15:44:42.964 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-28T15:44:42.964 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-28T15:44:42.964 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000199202 s, 2.6 MB/s 2024-03-28T15:44:42.965 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T15:44:43.021 DEBUG:teuthology.orchestra.run.smithi120:> stat /dev/vg_nvme/lv_4 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Access: 2024-03-28 15:44:20.395190365 +0000 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Modify: 2024-03-28 15:38:47.344595805 +0000 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout:Change: 2024-03-28 15:38:47.344595805 +0000 2024-03-28T15:44:43.076 INFO:teuthology.orchestra.run.smithi120.stdout: Birth: 2024-03-28 15:38:47.344595805 +0000 2024-03-28T15:44:43.077 DEBUG:teuthology.orchestra.run.smithi120:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T15:44:43.141 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records in 2024-03-28T15:44:43.142 INFO:teuthology.orchestra.run.smithi120.stderr:1+0 records out 2024-03-28T15:44:43.142 INFO:teuthology.orchestra.run.smithi120.stderr:512 bytes copied, 0.000169493 s, 3.0 MB/s 2024-03-28T15:44:43.143 DEBUG:teuthology.orchestra.run.smithi120:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T15:44:43.199 INFO:tasks.cephadm:Deploying osd.0 on smithi023 with /dev/vg_nvme/lv_4... 2024-03-28T15:44:43.199 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:44:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:43.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:43.437 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:43.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:43.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:43.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:44.389 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:44 smithi120 ceph-mon[36826]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:44.389 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:44 smithi120 ceph-mon[36826]: Deploying daemon mgr.z on smithi120 2024-03-28T15:44:44.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:44 smithi023 ceph-mon[32448]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:44.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:44 smithi023 ceph-mon[32448]: Deploying daemon mgr.z on smithi120 2024-03-28T15:44:44.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:44 smithi117 ceph-mon[36002]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:44.574 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:44 smithi117 ceph-mon[36002]: Deploying daemon mgr.z on smithi120 2024-03-28T15:44:44.768 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:44 smithi120 systemd[1]: Starting Ceph mgr.z for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:44:45.074 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:44 smithi120 podman[38671]: 2024-03-28 15:44:44.671773345 +0000 UTC m=+0.017574087 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:44:45.075 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:44 smithi120 podman[38671]: 2024-03-28 15:44:44.772012749 +0000 UTC m=+0.117813480 container create dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-28T15:44:45.075 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:44 smithi120 podman[38671]: 2024-03-28 15:44:44.990938715 +0000 UTC m=+0.336739459 container init dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T15:44:45.075 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:44 smithi120 podman[38671]: 2024-03-28 15:44:44.993407146 +0000 UTC m=+0.339207874 container start dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T15:44:45.075 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:45 smithi120 ceph-mgr[38690]: -- 172.21.15.120:0/1966118732 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eb859ea5a0 con 0x55eb859bec00 2024-03-28T15:44:45.075 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:45 smithi120 bash[38671]: dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f 2024-03-28T15:44:45.331 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:44:45.331 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:44:45.331 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T15:44:45.331 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T15:44:45.331 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133286 s, 787 MB/s 2024-03-28T15:44:45.332 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T15:44:45.339 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:45 smithi120 systemd[1]: Started Ceph mgr.z for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:44:45.839 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:45.532+0000 7f96ede18200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:44:45.839 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:45.625+0000 7f96ede18200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:44:45.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_4 2024-03-28T15:44:46.195 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:44:46.315 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:46.316 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:46 smithi023 ceph-mon[32448]: Deploying daemon mgr.y on smithi117 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:46 smithi117 ceph-mon[36002]: Deploying daemon mgr.y on smithi117 2024-03-28T15:44:46.404 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.265+0000 7f96ede18200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.359+0000 7f96ede18200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:46.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:46.406 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-mon[36826]: Deploying daemon mgr.y on smithi117 2024-03-28T15:44:46.693 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.507+0000 7f96ede18200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:44:46.693 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.606+0000 7f96ede18200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:44:46.693 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.692+0000 7f96ede18200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:44:47.257 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:46.962+0000 7f96ede18200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:44:47.257 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:47.073+0000 7f96ede18200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:44:47.257 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:47.160+0000 7f96ede18200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:44:47.257 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:47.256+0000 7f96ede18200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:44:47.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:47.435+0000 7f96ede18200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 -- 172.21.15.23:0/952964488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 msgr2=0x7feae0108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 --2- 172.21.15.23:0/952964488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae0108cb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fead800b0d0 tx=0x7fead800b3e0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 -- 172.21.15.23:0/952964488 shutdown_connections 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 --2- 172.21.15.23:0/952964488 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae0108cb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 --2- 172.21.15.23:0/952964488 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0100860 0x7feae0100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 --2- 172.21.15.23:0/952964488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0101a60 0x7feae0101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 -- 172.21.15.23:0/952964488 >> 172.21.15.23:0/952964488 conn(0x7feae00fbde0 msgr2=0x7feae00fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:44:47.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 -- 172.21.15.23:0/952964488 shutdown_connections 2024-03-28T15:44:47.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.620+0000 7feae811b700 1 -- 172.21.15.23:0/952964488 wait complete. 2024-03-28T15:44:47.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 Processor -- start 2024-03-28T15:44:47.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 -- start start 2024-03-28T15:44:47.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:47.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0101a60 0x7feae01977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:47.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae019e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:47.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae010ccd0 con 0x7feae0102400 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae010cb50 con 0x7feae0100860 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.621+0000 7feae811b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae010ce50 con 0x7feae0101a60 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae5eb7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae019e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae5eb7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:43338/0 (socket says 172.21.15.23:43338) 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae5eb7700 1 -- 172.21.15.23:0/467374624 learned_addr learned my addr 172.21.15.23:0/467374624 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae56b6700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0101a60 0x7feae01977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 -- 172.21.15.23:0/467374624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0101a60 msgr2=0x7feae01977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0101a60 0x7feae01977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 -- 172.21.15.23:0/467374624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 msgr2=0x7feae01972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 -- 172.21.15.23:0/467374624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae019ef10 con 0x7feae0102400 2024-03-28T15:44:47.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae5eb7700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae66b8700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae019e870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fead800b0d0 tx=0x7fead8007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8021070 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae019f1a0 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fead801db30 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead801bba0 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.622+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae019f650 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.623+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fead8024070 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.623+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae0068e50 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.623+0000 7fead2ffd700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 0x7feacc03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.623+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fead804d510 con 0x7feae0102400 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.623+0000 7feae5eb7700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 0x7feacc03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:44:47.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.624+0000 7feae5eb7700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 0x7feacc03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feadc0104e0 tx=0x7feadc00b450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:44:47.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.628+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fead8012db0 con 0x7feae0102400 2024-03-28T15:44:47.629 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:47 smithi117 bash[37362]: f9136aa59a52b65cb14ba8611967b4982786fc97b0b21ff0a7480c201709ddb4 2024-03-28T15:44:47.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:47.860+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7feae00625d0 con 0x7feacc039200 2024-03-28T15:44:48.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:47 smithi117 systemd[1]: Started Ceph mgr.y for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:44:48.359 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:48.088+0000 7f406460b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:48.185+0000 7f406460b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:48.360 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:44:48.434 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:44:48.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:48.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:48.732 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:48.433+0000 7f96ede18200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:44:48.733 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:48.578+0000 7f96ede18200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:44:49.006 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:48.917+0000 7f96ede18200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:44:49.006 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:49.005+0000 7f96ede18200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:44:49.074 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:48.865+0000 7f406460b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:44:49.074 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:48.957+0000 7f406460b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:44:49.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-mon[36826]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:49.339 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:49.218+0000 7f96ede18200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:44:49.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:49 smithi023 ceph-mon[32448]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:49.553 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-mon[36002]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:44:49.553 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.101+0000 7f406460b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:44:49.553 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.206+0000 7f406460b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:44:49.553 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.288+0000 7f406460b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:44:49.553 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.552+0000 7f406460b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:44:49.682 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:49.405+0000 7f96ede18200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:44:49.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.659+0000 7f406460b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:44:49.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.743+0000 7f406460b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:44:49.937 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:49.682+0000 7f96ede18200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:44:49.937 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:49.936+0000 7f96ede18200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:44:50.196 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:50.169+0000 7f96ede18200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:44:50.196 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:49.837+0000 7f406460b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:44:50.196 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:50.010+0000 7f406460b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:50.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:50.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:51.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:44:51.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T15:44:51.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]: dispatch 2024-03-28T15:44:51.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]': finished 2024-03-28T15:44:51.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:44:51.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:44:51.322 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:50.969+0000 7f406460b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:44:51.322 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:51.096+0000 7f406460b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:44:51.339 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:50.964+0000 7f96ede18200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:44:51.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:44:51.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T15:44:51.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]: dispatch 2024-03-28T15:44:51.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]': finished 2024-03-28T15:44:51.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:44:51.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: Reconfiguring mgr.x (unknown last config time)... 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: Reconfiguring daemon mgr.x on smithi023 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]: dispatch 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2658581382' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d73a4797-07a2-4f5c-9df2-ae9b5ec8c617"}]': finished 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: osdmap e5: 1 total, 0 up, 1 in 2024-03-28T15:44:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:44:51.711 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:51.427+0000 7f406460b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:44:51.711 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:51.516+0000 7f406460b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:44:51.711 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:51.711+0000 7f406460b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:44:51.837 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:51.528+0000 7f96ede18200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:44:51.837 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:51.623+0000 7f96ede18200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:44:51.838 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:44:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:44:51.837+0000 7f96ede18200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:44:52.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:51.892+0000 7f406460b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:44:52.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:52.220+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7fead801d650 con 0x7feae0102400 2024-03-28T15:44:52.434 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:52.174+0000 7f406460b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:52.434+0000 7f406460b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/243341915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: Standby manager daemon z started 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:44:52.435 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/243341915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: Standby manager daemon z started 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:44:52.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:52 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/243341915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: Standby manager daemon z started 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:44:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:52 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/2980869815' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:52.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:52.670+0000 7f406460b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:44:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:53 smithi117 ceph-mon[36002]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T15:44:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:44:53.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:53.463+0000 7f406460b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:44:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:53 smithi120 ceph-mon[36826]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T15:44:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:44:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:53 smithi023 ceph-mon[32448]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-28T15:44:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:44:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:54 smithi117 ceph-mon[36002]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:54.321 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:54.033+0000 7f406460b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:44:54.321 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:54.129+0000 7f406460b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:44:54.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:54 smithi120 ceph-mon[36826]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:54.612 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:54 smithi023 ceph-mon[32448]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:54.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:44:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:44:54.343+0000 7f406460b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:44:55.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:44:55.246+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fead802e350 con 0x7feae0102400 2024-03-28T15:44:55.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:55 smithi023 ceph-mon[32448]: Standby manager daemon y started 2024-03-28T15:44:55.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:55 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:44:55.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:55 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:55.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:55 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:44:55.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:55 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:55 smithi117 ceph-mon[36002]: Standby manager daemon y started 2024-03-28T15:44:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:55 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:44:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:55 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:55 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:44:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:55 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:55 smithi120 ceph-mon[36826]: Standby manager daemon y started 2024-03-28T15:44:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:55 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:44:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:55 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:44:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:55 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:44:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:55 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/1339588619' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:56.343 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:44:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:44:58.516 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:58 smithi023 ceph-mon[32448]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:58 smithi117 ceph-mon[36002]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:58 smithi120 ceph-mon[36826]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:44:59.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:44:59.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:44:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:44:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:44:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:44:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-28T15:44:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:44:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:00.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:00 smithi023 ceph-mon[32448]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:00.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:00 smithi023 ceph-mon[32448]: Deploying daemon osd.0 on smithi023 2024-03-28T15:45:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:00 smithi117 ceph-mon[36002]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:00 smithi117 ceph-mon[36002]: Deploying daemon osd.0 on smithi023 2024-03-28T15:45:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:00 smithi120 ceph-mon[36826]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:00 smithi120 ceph-mon[36826]: Deploying daemon osd.0 on smithi023 2024-03-28T15:45:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:02 smithi117 ceph-mon[36002]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:02 smithi120 ceph-mon[36826]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:02 smithi023 ceph-mon[32448]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:04 smithi117 ceph-mon[36002]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:04 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:04 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:04 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:04 smithi120 ceph-mon[36826]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:04 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:04 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:04 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:04.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:04 smithi023 ceph-mon[32448]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:04.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:04 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:04 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:04.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:04 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:06.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:06 smithi023 ceph-mon[32448]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:06.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:06 smithi117 ceph-mon[36002]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:06.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:06 smithi120 ceph-mon[36826]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:07.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:07.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:07.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.765+0000 7fead2ffd700 1 -- 172.21.15.23:0/467374624 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7feae00625d0 con 0x7feacc039200 2024-03-28T15:45:07.766 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 0 on host 'smithi023' 2024-03-28T15:45:07.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 msgr2=0x7feacc03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 0x7feacc03b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feadc0104e0 tx=0x7feadc00b450 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 msgr2=0x7feae019e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae019e870 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fead800b0d0 tx=0x7fead8007a20 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 shutdown_connections 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feacc039200 0x7feacc03b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feae0102400 0x7feae019e870 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feae0100860 0x7feae01972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 --2- 172.21.15.23:0/467374624 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feae0101a60 0x7feae01977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.767+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 >> 172.21.15.23:0/467374624 conn(0x7feae00fbde0 msgr2=0x7feae0104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.768+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 shutdown_connections 2024-03-28T15:45:07.768 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:07.768+0000 7feae811b700 1 -- 172.21.15.23:0/467374624 wait complete. 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:08.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.449 DEBUG:teuthology.orchestra.run.smithi023:osd.0> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.0.service 2024-03-28T15:45:08.451 INFO:tasks.cephadm:Deploying osd.1 on smithi117 with /dev/vg_nvme/lv_4... 2024-03-28T15:45:08.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:45:08.694 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:45:08.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:08 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:08 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:08.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:08 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:08.998 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:08 smithi023 ceph-mon[32448]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:45:09.045 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:08 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:09.045 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:08 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:09.045 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:08 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:09.045 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:08 smithi117 ceph-mon[36002]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:45:09.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:08 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:09.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:08 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:09.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:08 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:09.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:08 smithi120 ceph-mon[36826]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-28T15:45:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:09 smithi023 ceph-mon[32448]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:09 smithi023 ceph-mon[32448]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:45:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:09 smithi023 ceph-mon[32448]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:45:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:09 smithi023 ceph-mon[32448]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:45:09.839 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:09.929 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:09 smithi117 ceph-mon[36002]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:09.929 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:09 smithi117 ceph-mon[36002]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:45:09.929 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:09 smithi117 ceph-mon[36002]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:45:09.929 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:09 smithi117 ceph-mon[36002]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:45:09.929 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:09 smithi120 ceph-mon[36826]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:09 smithi120 ceph-mon[36826]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-28T15:45:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:09 smithi120 ceph-mon[36826]: osdmap e6: 1 total, 0 up, 1 in 2024-03-28T15:45:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:09 smithi120 ceph-mon[36826]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:45:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:45:10.480 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:45:10.481 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-28T15:45:10.481 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-28T15:45:10.481 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110778 s, 947 MB/s 2024-03-28T15:45:10.481 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-28T15:45:10.670 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0[47228]: 2024-03-28T15:45:10.183+0000 7f215ca04700 -1 osd.0 0 waiting for initial osdmap 2024-03-28T15:45:10.670 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0[47228]: 2024-03-28T15:45:10.187+0000 7f2153fee700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:45:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-mon[32448]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:45:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-mon[32448]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:45:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.007 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.008 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:10 smithi023 ceph-mon[32448]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' 2024-03-28T15:45:11.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-03-28T15:45:11.016 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:10 smithi117 ceph-mon[36002]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:45:11.016 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:10 smithi117 ceph-mon[36002]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:45:11.016 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.016 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.016 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:10 smithi117 ceph-mon[36002]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' 2024-03-28T15:45:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:10 smithi120 ceph-mon[36826]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:45:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:10 smithi120 ceph-mon[36826]: osdmap e7: 1 total, 0 up, 1 in 2024-03-28T15:45:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:10 smithi120 ceph-mon[36826]: from='osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101]' entity='osd.0' 2024-03-28T15:45:11.261 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:45:12.035 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:45:12.035 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] boot 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:12.036 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] boot 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:12.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:12.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:12.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: osd.0 [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] boot 2024-03-28T15:45:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: osdmap e8: 1 total, 1 up, 1 in 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:12.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:12.751 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 -- 172.21.15.117:0/1833591047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1022c0 msgr2=0x7f2d8c102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 --2- 172.21.15.117:0/1833591047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c102720 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0099c0 tx=0x7f2d7c009cd0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 -- 172.21.15.117:0/1833591047 shutdown_connections 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 --2- 172.21.15.117:0/1833591047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 0x7f2d8c109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 --2- 172.21.15.117:0/1833591047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c1014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 --2- 172.21.15.117:0/1833591047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c102720 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 -- 172.21.15.117:0/1833591047 >> 172.21.15.117:0/1833591047 conn(0x7f2d8c0fc660 msgr2=0x7f2d8c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 -- 172.21.15.117:0/1833591047 shutdown_connections 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.749+0000 7f2d94717700 1 -- 172.21.15.117:0/1833591047 wait complete. 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 Processor -- start 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 -- start start 2024-03-28T15:45:12.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c19d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 0x7f2d8c19f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c10cd00 con 0x7f2d8c1022c0 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c10cb80 con 0x7f2d8c1010c0 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c10ce80 con 0x7f2d8c102c60 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:47408/0 (socket says 172.21.15.117:47408) 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 -- 172.21.15.117:0/2810479244 learned_addr learned my addr 172.21.15.117:0/2810479244 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d92cb4700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 0x7f2d8c19f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d91cb2700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c19d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 -- 172.21.15.117:0/2810479244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 msgr2=0x7f2d8c19f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 0x7f2d8c19f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 -- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 msgr2=0x7f2d8c19d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:12.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c19d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 -- 172.21.15.117:0/2810479244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c1a00c0 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d91cb2700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c19d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d924b3700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d88007f10 tx=0x7f2d8800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8800dc90 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d8800de30 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d880133f0 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.750+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c1a03b0 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.751+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c1a0940 con 0x7f2d8c1010c0 2024-03-28T15:45:12.754 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.752+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c068e50 con 0x7f2d8c1010c0 2024-03-28T15:45:12.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.752+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f2d8801a020 con 0x7f2d8c1010c0 2024-03-28T15:45:12.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.753+0000 7f2d837fe700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 0x7f2d780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:12.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.753+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f2d880cc710 con 0x7f2d8c1010c0 2024-03-28T15:45:12.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.753+0000 7f2d91cb2700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 0x7f2d780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:12.755 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.754+0000 7f2d91cb2700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 0x7f2d780a4d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0039c0 tx=0x7f2d7c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:12.757 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.758+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2d88010070 con 0x7f2d8c1010c0 2024-03-28T15:45:12.963 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:12.963+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d8c0625d0 con 0x7f2d780a2880 2024-03-28T15:45:12.963 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:12 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi023 2024-03-28T15:45:12.963 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:12 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T15:45:13.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:12 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi023 2024-03-28T15:45:13.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:12 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T15:45:13.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:12 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi023 2024-03-28T15:45:13.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:12 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 17088M 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:14.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: osdmap e9: 1 total, 1 up, 1 in 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:14.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/1300897611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]': finished 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:45:16.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/1300897611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]': finished 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:45:16.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:16.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:16.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/1300897611' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]: dispatch 2024-03-28T15:45:16.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6fe2a5d7-6406-4527-bf36-1ad15e345a54"}]': finished 2024-03-28T15:45:16.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: osdmap e10: 2 total, 1 up, 2 in 2024-03-28T15:45:16.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:17.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:16 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/3549112692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:17.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:16 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/3549112692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:17.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:16 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/3549112692' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:18.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:17 smithi117 ceph-mon[36002]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:18.081 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:17 smithi120 ceph-mon[36826]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:18.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:17 smithi023 ceph-mon[32448]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:20.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:19 smithi117 ceph-mon[36002]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:20.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:19 smithi120 ceph-mon[36826]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:19 smithi023 ceph-mon[32448]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:22.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:21 smithi117 ceph-mon[36002]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:22.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:21 smithi120 ceph-mon[36826]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:22.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:21 smithi023 ceph-mon[32448]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:23.057 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:45:23.058 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:23.058 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:22 smithi117 ceph-mon[36002]: Deploying daemon osd.1 on smithi117 2024-03-28T15:45:23.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:45:23.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:23.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:22 smithi120 ceph-mon[36826]: Deploying daemon osd.1 on smithi117 2024-03-28T15:45:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-28T15:45:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:22 smithi023 ceph-mon[32448]: Deploying daemon osd.1 on smithi117 2024-03-28T15:45:23.944 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:23 smithi117 ceph-mon[36002]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:24.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:23 smithi120 ceph-mon[36826]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:24.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:23 smithi023 ceph-mon[32448]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:26.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:25 smithi117 ceph-mon[36002]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:26.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:25 smithi120 ceph-mon[36826]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:25 smithi023 ceph-mon[32448]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:27.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:27 smithi120 ceph-mon[36826]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:27.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:27.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:27.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:27.970 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:27 smithi117 ceph-mon[36002]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:27.970 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:27 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:27.970 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:27 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:27.970 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:27 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:27 smithi023 ceph-mon[32448]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:30.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:29 smithi120 ceph-mon[36826]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:30.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:29 smithi023 ceph-mon[32448]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:30.207 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:29 smithi117 ceph-mon[36002]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:30.725 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 1 on host 'smithi117' 2024-03-28T15:45:30.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.725+0000 7f2d837fe700 1 -- 172.21.15.117:0/2810479244 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2d8c0625d0 con 0x7f2d780a2880 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.726+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 msgr2=0x7f2d780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.726+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 0x7f2d780a4d40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c0039c0 tx=0x7f2d7c000bc0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.726+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 msgr2=0x7f2d8c06a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.726+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2d88007f10 tx=0x7f2d8800d3b0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 shutdown_connections 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2d780a2880 0x7f2d780a4d40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2d8c102c60 0x7f2d8c19f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c1010c0 0x7f2d8c06a9a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 --2- 172.21.15.117:0/2810479244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2d8c1022c0 0x7f2d8c19d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 >> 172.21.15.117:0/2810479244 conn(0x7f2d8c0fc660 msgr2=0x7f2d8c105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 shutdown_connections 2024-03-28T15:45:30.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:45:30.727+0000 7f2d94717700 1 -- 172.21.15.117:0/2810479244 wait complete. 2024-03-28T15:45:31.037 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.037 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.038 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:31.336 DEBUG:teuthology.orchestra.run.smithi117:osd.1> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.1.service 2024-03-28T15:45:31.338 INFO:tasks.cephadm:Deploying osd.2 on smithi120 with /dev/vg_nvme/lv_4... 2024-03-28T15:45:31.338 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-28T15:45:31.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:31.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:31.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:31.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:31.581 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:45:32.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:31 smithi117 ceph-mon[36002]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:32.455 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:31 smithi117 ceph-mon[36002]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:32.455 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:31 smithi117 ceph-mon[36002]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:32.455 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:31 smithi023 ceph-mon[32448]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:32.456 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:31 smithi023 ceph-mon[32448]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:32.456 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:31 smithi023 ceph-mon[32448]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:32.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:31 smithi120 ceph-mon[36826]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:32.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:31 smithi120 ceph-mon[36826]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:32.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:31 smithi120 ceph-mon[36826]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-28T15:45:33.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-mon[36002]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:45:33.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-mon[36002]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-mon[36002]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:45:33.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:33.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-mon[36002]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:33 smithi120 ceph-mon[36826]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:45:33.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:33 smithi120 ceph-mon[36826]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:33 smithi120 ceph-mon[36826]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:45:33.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:33.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:33 smithi120 ceph-mon[36826]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111173 s, 943 MB/s 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 2024-03-28T15:45:33.372 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-28T15:45:33.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:33 smithi023 ceph-mon[32448]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-28T15:45:33.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:33 smithi023 ceph-mon[32448]: from='osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:33 smithi023 ceph-mon[32448]: osdmap e11: 2 total, 1 up, 2 in 2024-03-28T15:45:33.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:33.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:33 smithi023 ceph-mon[32448]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:45:33.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1[40568]: 2024-03-28T15:45:33.501+0000 7f1d5311f700 -1 osd.1 0 waiting for initial osdmap 2024-03-28T15:45:33.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:45:33 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1[40568]: 2024-03-28T15:45:33.504+0000 7f1d4a709700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:45:34.004 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_4 2024-03-28T15:45:34.237 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: from='osd.1 ' entity='osd.1' 2024-03-28T15:45:34.242 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: from='osd.1 ' entity='osd.1' 2024-03-28T15:45:34.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: osdmap e12: 2 total, 1 up, 2 in 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: from='osd.1 ' entity='osd.1' 2024-03-28T15:45:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:35.307 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] boot 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:45:35.308 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:35.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] boot 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:45:35.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: osd.1 [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] boot 2024-03-28T15:45:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: osdmap e13: 2 total, 2 up, 2 in 2024-03-28T15:45:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 -- 172.21.15.120:0/3278124621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec074b70 msgr2=0x7f75ec074ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 --2- 172.21.15.120:0/3278124621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec074b70 0x7f75ec074ff0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f75e0009990 tx=0x7f75e0009ca0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 -- 172.21.15.120:0/3278124621 shutdown_connections 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 --2- 172.21.15.120:0/3278124621 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 0x7f75ec10c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 --2- 172.21.15.120:0/3278124621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec076140 0x7f75ec0745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 --2- 172.21.15.120:0/3278124621 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec074b70 0x7f75ec074ff0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.789 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 -- 172.21.15.120:0/3278124621 >> 172.21.15.120:0/3278124621 conn(0x7f75ec0fd7a0 msgr2=0x7f75ec0ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:35.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 -- 172.21.15.120:0/3278124621 shutdown_connections 2024-03-28T15:45:35.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.788+0000 7f75f1df1700 1 -- 172.21.15.120:0/3278124621 wait complete. 2024-03-28T15:45:35.790 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 Processor -- start 2024-03-28T15:45:35.791 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 -- start start 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec074b70 0x7f75ec19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 0x7f75ec1a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec10ffb0 con 0x7f75ec10a0e0 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec10fe30 con 0x7f75ec074b70 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75f1df1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ec110130 con 0x7f75ec076140 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:48608/0 (socket says 172.21.15.120:48608) 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 -- 172.21.15.120:0/1863375169 learned_addr learned my addr 172.21.15.120:0/1863375169 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eb7fe700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec074b70 0x7f75ec19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75ebfff700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 0x7f75ec1a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 -- 172.21.15.120:0/1863375169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec074b70 msgr2=0x7f75ec19a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec074b70 0x7f75ec19a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 -- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 msgr2=0x7f75ec1a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 0x7f75ec1a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.789+0000 7f75eaffd700 1 -- 172.21.15.120:0/1863375169 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ec1a2210 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75eaffd700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75e0004790 tx=0x7f75e0009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e0027070 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75e0004210 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e000f460 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ec1a24a0 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.790+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ec1a28f0 con 0x7f75ec076140 2024-03-28T15:45:35.793 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.791+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ec068e50 con 0x7f75ec076140 2024-03-28T15:45:35.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.792+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f75e002a070 con 0x7f75ec076140 2024-03-28T15:45:35.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.792+0000 7f75e8ff9700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 0x7f75d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:35.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.793+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f75e00ce000 con 0x7f75ec076140 2024-03-28T15:45:35.794 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.793+0000 7f75eb7fe700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 0x7f75d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:35.795 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.793+0000 7f75eb7fe700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 0x7f75d40a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75e40060b0 tx=0x7f75e4007790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:35.797 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:35.797+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f75e0013070 con 0x7f75ec076140 2024-03-28T15:45:36.005 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:36.004+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f75ec0625d0 con 0x7f75d40a2800 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi117 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:36.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi117 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi117 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 17080M 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:45:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:37.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:37 smithi117 ceph-mon[36002]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:37.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:37 smithi117 ceph-mon[36002]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:45:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:37 smithi120 ceph-mon[36826]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:37 smithi120 ceph-mon[36826]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:45:37.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:37 smithi023 ceph-mon[32448]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:45:37.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:37 smithi023 ceph-mon[32448]: osdmap e14: 2 total, 2 up, 2 in 2024-03-28T15:45:38.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:38 smithi117 ceph-mon[36002]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:38 smithi120 ceph-mon[36826]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:38.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:38 smithi023 ceph-mon[32448]: pgmap v65: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:39.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:39 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/1954256583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:39 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:39 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]': finished 2024-03-28T15:45:39.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:39 smithi117 ceph-mon[36002]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:45:39.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:39 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/1954256583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:39 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:39 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]': finished 2024-03-28T15:45:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:39 smithi120 ceph-mon[36826]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:45:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:39.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:39 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1954256583' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:39 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]: dispatch 2024-03-28T15:45:39.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:39 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9c0639e5-8825-479c-a148-f3a525d6995c"}]': finished 2024-03-28T15:45:39.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:39 smithi023 ceph-mon[32448]: osdmap e15: 3 total, 2 up, 3 in 2024-03-28T15:45:39.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:40.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:40 smithi117 ceph-mon[36002]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:40.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:40 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/528770981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:40 smithi120 ceph-mon[36826]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:40 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/528770981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:40.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:40 smithi023 ceph-mon[32448]: pgmap v66: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:40.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:40 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/528770981' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:45:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:42 smithi117 ceph-mon[36002]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:42.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:42 smithi120 ceph-mon[36826]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:42.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:42 smithi023 ceph-mon[32448]: pgmap v68: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:44.229 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:44 smithi120 ceph-mon[36826]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:44.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:44 smithi117 ceph-mon[36002]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:44.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:44 smithi023 ceph-mon[32448]: pgmap v69: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:46 smithi117 ceph-mon[36002]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:45:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:46.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:46 smithi120 ceph-mon[36826]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:46.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:45:46.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:46 smithi023 ceph-mon[32448]: pgmap v70: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-28T15:45:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:47.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:47 smithi117 ceph-mon[36002]: Deploying daemon osd.2 on smithi120 2024-03-28T15:45:47.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:47 smithi120 ceph-mon[36826]: Deploying daemon osd.2 on smithi120 2024-03-28T15:45:47.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:47 smithi023 ceph-mon[32448]: Deploying daemon osd.2 on smithi120 2024-03-28T15:45:48.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:48 smithi117 ceph-mon[36002]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:48.321 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:48 smithi120 ceph-mon[36826]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:48 smithi023 ceph-mon[32448]: pgmap v71: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:50.154 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:50 smithi120 ceph-mon[36826]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:50.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:50 smithi117 ceph-mon[36002]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:50.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:50 smithi023 ceph-mon[32448]: pgmap v72: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:52.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:51 smithi117 ceph-mon[36002]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:52.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:52.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:51 smithi120 ceph-mon[36826]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:52.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:52.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:51 smithi023 ceph-mon[32448]: pgmap v73: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:52.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:52.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:54.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: pgmap v74: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:54.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:54.272 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.271+0000 7f75e8ff9700 1 -- 172.21.15.120:0/1863375169 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75ec0625d0 con 0x7f75d40a2800 2024-03-28T15:45:54.272 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 2 on host 'smithi120' 2024-03-28T15:45:54.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.273+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 msgr2=0x7f75d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:54.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.273+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 0x7f75d40a4cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75e40060b0 tx=0x7f75e4007790 comp rx=0 tx=0).stop 2024-03-28T15:45:54.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.273+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 msgr2=0x7f75ec19aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:54.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.273+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f75e0004790 tx=0x7f75e0009580 comp rx=0 tx=0).stop 2024-03-28T15:45:54.274 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 shutdown_connections 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f75d40a2800 0x7f75d40a4cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f75ec10a0e0 0x7f75ec1a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75ec074b70 0x7f75ec19a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 --2- 172.21.15.120:0/1863375169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f75ec076140 0x7f75ec19aaf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 >> 172.21.15.120:0/1863375169 conn(0x7f75ec0fd7a0 msgr2=0x7f75ec106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 shutdown_connections 2024-03-28T15:45:54.275 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:45:54.274+0000 7f75f1df1700 1 -- 172.21.15.120:0/1863375169 wait complete. 2024-03-28T15:45:54.885 DEBUG:teuthology.orchestra.run.smithi120:osd.2> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.2.service 2024-03-28T15:45:54.887 INFO:tasks.cephadm:Deploying osd.3 on smithi023 with /dev/vg_nvme/lv_3... 2024-03-28T15:45:54.888 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:45:55.139 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:45:55.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:55.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:55 smithi023 ceph-mon[32448]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:55.503 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:55 smithi023 ceph-mon[32448]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:55 smithi117 ceph-mon[36002]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:55 smithi117 ceph-mon[36002]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:45:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:55 smithi120 ceph-mon[36826]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:55 smithi120 ceph-mon[36826]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:56.325 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-mon[36826]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:56.326 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2[41989]: 2024-03-28T15:45:56.251+0000 7f38ff2dd700 -1 osd.2 0 waiting for initial osdmap 2024-03-28T15:45:56.326 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:45:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2[41989]: 2024-03-28T15:45:56.256+0000 7f38fb0d0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:56.332 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:56 smithi023 ceph-mon[32448]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: pgmap v75: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: osdmap e16: 3 total, 2 up, 3 in 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: from='osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:56 smithi117 ceph-mon[36002]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:45:57.212 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:45:57.213 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:45:57.213 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T15:45:57.213 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T15:45:57.213 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0165847 s, 632 MB/s 2024-03-28T15:45:57.213 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T15:45:57.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:57 smithi023 ceph-mon[32448]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:45:57.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:57 smithi023 ceph-mon[32448]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:45:57.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:57.515 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:57.540 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:57 smithi120 ceph-mon[36826]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:45:57.540 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:57 smithi120 ceph-mon[36826]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:45:57.541 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:57.541 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:57.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:57 smithi117 ceph-mon[36002]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:45:57.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:57 smithi117 ceph-mon[36002]: osdmap e17: 3 total, 2 up, 3 in 2024-03-28T15:45:57.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:57.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.033 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_3 2024-03-28T15:45:58.278 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:45:58.520 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] boot 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:58.521 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:58 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:45:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:45:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] boot 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:58.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:58 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:45:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:45:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: osd.2 [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] boot 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: osdmap e18: 3 total, 3 up, 3 in 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:45:58.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:58 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:45:59.412 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:59 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi120 2024-03-28T15:45:59.413 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:59 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-28T15:45:59.413 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:59 smithi023 ceph-mon[32448]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:45:59.413 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:45:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:45:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:59 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi120 2024-03-28T15:45:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:59 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-28T15:45:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:59 smithi117 ceph-mon[36002]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:45:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:45:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:45:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:59 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi120 2024-03-28T15:45:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:59 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 17080M 2024-03-28T15:45:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:59 smithi120 ceph-mon[36826]: osdmap e19: 3 total, 3 up, 3 in 2024-03-28T15:45:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:45:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/1893603030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102cd0 msgr2=0x7fe6a8109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/1893603030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102cd0 0x7fe6a8109580 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe6a00099c0 tx=0x7fe6a0009cd0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/1893603030 shutdown_connections 2024-03-28T15:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/1893603030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102cd0 0x7fe6a8109580 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/1893603030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 0x7fe6a8101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/1893603030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102330 0x7fe6a8102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/1893603030 >> 172.21.15.23:0/1893603030 conn(0x7fe6a80fc6f0 msgr2=0x7fe6a80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/1893603030 shutdown_connections 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/1893603030 wait complete. 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.963+0000 7fe6ad1f1700 1 Processor -- start 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 -- start start 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 0x7fe6a8197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 0x7fe6a819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a810ccb0 con 0x7fe6a8102330 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a810cb30 con 0x7fe6a8102cd0 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6ad1f1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a810ce30 con 0x7fe6a8101130 2024-03-28T15:45:59.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43198/0 (socket says 172.21.15.23:43198) 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 -- 172.21.15.23:0/2799038603 learned_addr learned my addr 172.21.15.23:0/2799038603 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a6d9d700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 0x7fe6a8197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a759e700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 0x7fe6a819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 -- 172.21.15.23:0/2799038603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 msgr2=0x7fe6a8197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 0x7fe6a8197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 -- 172.21.15.23:0/2799038603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 msgr2=0x7fe6a819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 0x7fe6a819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 -- 172.21.15.23:0/2799038603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a819eed0 con 0x7fe6a8102330 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a759e700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 0x7fe6a819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.964+0000 7fe6a659c700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6900099a0 tx=0x7fe690009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.965+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69001b070 con 0x7fe6a8102330 2024-03-28T15:45:59.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.965+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a806a8f0 con 0x7fe6a8102330 2024-03-28T15:45:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.965+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe69000d260 con 0x7fe6a8102330 2024-03-28T15:45:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.965+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69000caa0 con 0x7fe6a8102330 2024-03-28T15:45:59.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.965+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a806ae30 con 0x7fe6a8102330 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.966+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a8068e50 con 0x7fe6a8102330 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.967+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fe69001e070 con 0x7fe6a8102330 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.967+0000 7fe69ffff700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 0x7fe6940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.968+0000 7fe6a6d9d700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 0x7fe6940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.968+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3226+0+0 (secure 0 0 0) 0x7fe6900ce420 con 0x7fe6a8102330 2024-03-28T15:45:59.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.968+0000 7fe6a6d9d700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 0x7fe6940a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8102190 tx=0x7fe698009180 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:45:59.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:45:59.971+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe69009c980 con 0x7fe6a8102330 2024-03-28T15:46:00.185 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:00.184+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6a806b150 con 0x7fe6940a2800 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: pgmap v81: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: osdmap e20: 3 total, 3 up, 3 in 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:00.659 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:00.660 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:00.660 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51755]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:00.916 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51755]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:00.916 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51755]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:00.916 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51755]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:01.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51799]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:01.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51799]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:01.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51799]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:01.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 15:46:00 smithi023 sudo[51799]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:01.451 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44284]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:01.451 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44284]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:01.451 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44284]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:01.451 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44284]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:01.451 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:01 smithi117 ceph-mon[36002]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:01.451 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:01 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:01.451 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:01 smithi117 ceph-mon[36002]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:46:01.452 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:01 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:01.452 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:01 smithi117 ceph-mon[36002]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:46:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:01 smithi120 ceph-mon[36826]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:01 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:01 smithi120 ceph-mon[36826]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:46:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:01 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:01 smithi120 ceph-mon[36826]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:46:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:01 smithi023 ceph-mon[32448]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:01 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-28T15:46:01.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:01 smithi023 ceph-mon[32448]: osdmap e21: 3 total, 3 up, 3 in 2024-03-28T15:46:01.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:01 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:01.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:01 smithi023 ceph-mon[32448]: osdmap e22: 3 total, 3 up, 3 in 2024-03-28T15:46:01.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44287]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:01.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44287]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:01.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44287]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:01.821 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 15:46:01 smithi117 sudo[44287]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:01.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:01.963+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe6900988f0 con 0x7fe6a8102330 2024-03-28T15:46:01.973 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45621]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:01.973 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45621]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:01.973 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45621]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:01.973 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45621]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:02.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45624]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:02.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45624]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:02.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45624]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:02.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 15:46:01 smithi120 sudo[45624]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51848]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51848]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51848]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51848]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 ceph-mon[36826]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 ceph-mon[36826]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 sudo[45627]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 sudo[45627]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:02.789 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 sudo[45627]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:02.790 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:02 smithi120 sudo[45627]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:02 smithi117 ceph-mon[36002]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:02 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:02 smithi117 ceph-mon[36002]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T15:46:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:02 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:02 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51864]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51864]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51864]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 sudo[51864]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 ceph-mon[32448]: pgmap v84: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 ceph-mon[32448]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-28T15:46:02.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:02.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:02 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 sudo[45630]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:03.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 sudo[45630]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:03.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 sudo[45630]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:03.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 sudo[45630]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:03.526 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-28T15:46:03.527 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:03.527 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44290]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:03.527 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44290]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]': finished 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:03 smithi023 ceph-mon[32448]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44293]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44293]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44293]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 sudo[44293]: pam_unix(sudo:session): session closed for user root 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]: dispatch 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.823 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]': finished 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:03 smithi117 ceph-mon[36002]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/447919999' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5757ad6-b03a-4985-ba33-e0010a2c3161"}]': finished 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: osdmap e23: 4 total, 3 up, 4 in 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-28T15:46:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:46:03.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:46:03.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:46:03.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:03 smithi120 ceph-mon[36826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-28T15:46:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:04 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2915426586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:04 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2915426586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:04.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:04 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2915426586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:05 smithi117 ceph-mon[36002]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:05 smithi120 ceph-mon[36826]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:05.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:05 smithi023 ceph-mon[32448]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:07 smithi023 ceph-mon[32448]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:07.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:07 smithi117 ceph-mon[36002]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:07 smithi120 ceph-mon[36826]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:09 smithi117 ceph-mon[36002]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:09 smithi120 ceph-mon[36826]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:09 smithi023 ceph-mon[32448]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:46:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:46:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-28T15:46:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:11.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:11 smithi023 ceph-mon[32448]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:11.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:11 smithi023 ceph-mon[32448]: Deploying daemon osd.3 on smithi023 2024-03-28T15:46:11.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:11 smithi117 ceph-mon[36002]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:11.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:11 smithi117 ceph-mon[36002]: Deploying daemon osd.3 on smithi023 2024-03-28T15:46:11.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:11 smithi120 ceph-mon[36826]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:11.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:11 smithi120 ceph-mon[36826]: Deploying daemon osd.3 on smithi023 2024-03-28T15:46:13.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:13 smithi023 ceph-mon[32448]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:13 smithi117 ceph-mon[36002]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:13.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:13 smithi120 ceph-mon[36826]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:15.766 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:15 smithi023 ceph-mon[32448]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:15 smithi117 ceph-mon[36002]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:15 smithi120 ceph-mon[36826]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:17.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:17.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:17.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:17.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:18.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:18 smithi120 ceph-mon[36826]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:18.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:18 smithi117 ceph-mon[36002]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:18.340 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:18 smithi023 ceph-mon[32448]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:19.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:19.894 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 3 on host 'smithi023' 2024-03-28T15:46:19.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.893+0000 7fe69ffff700 1 -- 172.21.15.23:0/2799038603 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6a806b150 con 0x7fe6940a2800 2024-03-28T15:46:19.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 msgr2=0x7fe6940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:19.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 0x7fe6940a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe6a8102190 tx=0x7fe698009180 comp rx=0 tx=0).stop 2024-03-28T15:46:19.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 msgr2=0x7fe6a81977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:19.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe6900099a0 tx=0x7fe690009cb0 comp rx=0 tx=0).stop 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 shutdown_connections 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6940a2800 0x7fe6940a4cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6a8102cd0 0x7fe6a819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe6a8101130 0x7fe6a8197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 --2- 172.21.15.23:0/2799038603 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe6a8102330 0x7fe6a81977b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 >> 172.21.15.23:0/2799038603 conn(0x7fe6a80fc6f0 msgr2=0x7fe6a81055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 shutdown_connections 2024-03-28T15:46:19.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:46:19.896+0000 7fe6ad1f1700 1 -- 172.21.15.23:0/2799038603 wait complete. 2024-03-28T15:46:20.585 DEBUG:teuthology.orchestra.run.smithi023:osd.3> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.3.service 2024-03-28T15:46:20.588 INFO:tasks.cephadm:Deploying osd.4 on smithi117 with /dev/vg_nvme/lv_3... 2024-03-28T15:46:20.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:46:20.833 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:46:20.955 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:20.955 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:20.955 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:20.955 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:20 smithi117 ceph-mon[36002]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:46:21.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:21.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:21.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:21.114 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:20 smithi023 ceph-mon[32448]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:46:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:20 smithi120 ceph-mon[36826]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-28T15:46:22.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:21 smithi023 ceph-mon[32448]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:22.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:21 smithi023 ceph-mon[32448]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:46:22.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:21 smithi023 ceph-mon[32448]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:46:22.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:21 smithi023 ceph-mon[32448]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:46:22.057 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:21 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:22.167 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:21 smithi117 ceph-mon[36002]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:22.167 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:21 smithi117 ceph-mon[36002]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:46:22.167 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:21 smithi117 ceph-mon[36002]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:46:22.167 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:21 smithi117 ceph-mon[36002]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:46:22.167 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:21 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:21 smithi120 ceph-mon[36826]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:21 smithi120 ceph-mon[36826]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-28T15:46:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:21 smithi120 ceph-mon[36826]: osdmap e24: 4 total, 3 up, 4 in 2024-03-28T15:46:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:21 smithi120 ceph-mon[36826]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:46:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:21 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:22.670 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T15:46:22.316+0000 7fc63cb9a700 -1 osd.3 0 waiting for initial osdmap 2024-03-28T15:46:22.670 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T15:46:22.322+0000 7fc634184700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011251 s, 932 MB/s 2024-03-28T15:46:22.736 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-28T15:46:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-mon[32448]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:46:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-mon[32448]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:46:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:22 smithi023 ceph-mon[32448]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' 2024-03-28T15:46:23.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:22 smithi117 ceph-mon[36002]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:46:23.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:22 smithi117 ceph-mon[36002]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:46:23.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.283 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.283 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:22 smithi117 ceph-mon[36002]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' 2024-03-28T15:46:23.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-03-28T15:46:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:22 smithi120 ceph-mon[36826]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:46:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:22 smithi120 ceph-mon[36826]: osdmap e25: 4 total, 3 up, 4 in 2024-03-28T15:46:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:22 smithi120 ceph-mon[36826]: from='osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629]' entity='osd.3' 2024-03-28T15:46:23.563 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:46:24.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:46:24.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:46:24.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] boot 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:24.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:46:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:46:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] boot 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:24.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: osd.3 [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] boot 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: osdmap e26: 4 total, 4 up, 4 in 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:46:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:24.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:24.953 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:24 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi023 2024-03-28T15:46:24.953 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:24 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T15:46:24.953 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:24 smithi117 ceph-mon[36002]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:46:25.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:24 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi023 2024-03-28T15:46:25.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:24 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T15:46:25.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:24 smithi023 ceph-mon[32448]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- 172.21.15.117:0/2421865004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 msgr2=0x7f5f981025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 --2- 172.21.15.117:0/2421865004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f981025e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5f940099c0 tx=0x7f5f94009cd0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- 172.21.15.117:0/2421865004 shutdown_connections 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 --2- 172.21.15.117:0/2421865004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 0x7f5f9810a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 --2- 172.21.15.117:0/2421865004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f981025e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 --2- 172.21.15.117:0/2421865004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 0x7f5f98103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- 172.21.15.117:0/2421865004 >> 172.21.15.117:0/2421865004 conn(0x7f5f980fd7a0 msgr2=0x7f5f980ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:46:25.186 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- 172.21.15.117:0/2421865004 shutdown_connections 2024-03-28T15:46:25.187 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- 172.21.15.117:0/2421865004 wait complete. 2024-03-28T15:46:25.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 Processor -- start 2024-03-28T15:46:25.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.185+0000 7f5f9f701700 1 -- start start 2024-03-28T15:46:25.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:25.188 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 0x7f5f981989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 0x7f5f9819fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9810de10 con 0x7f5f98103d60 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9810dc90 con 0x7f5f981021c0 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9810df90 con 0x7f5f981033c0 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:48970/0 (socket says 172.21.15.117:48970) 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 -- 172.21.15.117:0/4226227313 learned_addr learned my addr 172.21.15.117:0/4226227313 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9dc9e700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 0x7f5f9819fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9cc9c700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 0x7f5f981989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 -- 172.21.15.117:0/4226227313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 msgr2=0x7f5f981989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 0x7f5f981989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 -- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 msgr2=0x7f5f9819fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:25.189 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 0x7f5f9819fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 -- 172.21.15.117:0/4226227313 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f981a00e0 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9d49d700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098e0 tx=0x7f5f94009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94027070 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f981a0370 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f94004150 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f94022e60 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.186+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f981a0820 con 0x7f5f981021c0 2024-03-28T15:46:25.190 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.187+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f98068e50 con 0x7f5f981021c0 2024-03-28T15:46:25.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.189+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5f9402a070 con 0x7f5f981021c0 2024-03-28T15:46:25.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.189+0000 7f5f8e7fc700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 0x7f5f840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:25.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.189+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f5f940ce6d0 con 0x7f5f981021c0 2024-03-28T15:46:25.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.190+0000 7f5f9cc9c700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 0x7f5f840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:25.191 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.190+0000 7f5f9cc9c700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 0x7f5f840a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009c10 tx=0x7f5f90009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:46:25.193 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.193+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f94013070 con 0x7f5f981021c0 2024-03-28T15:46:25.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:24 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi023 2024-03-28T15:46:25.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:24 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 8544M 2024-03-28T15:46:25.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:24 smithi120 ceph-mon[36826]: osdmap e27: 4 total, 4 up, 4 in 2024-03-28T15:46:25.398 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:25.397+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f980625d0 con 0x7f5f840a2890 2024-03-28T15:46:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:25 smithi023 ceph-mon[32448]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:25 smithi023 ceph-mon[32448]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:46:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:26.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:26.186 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:25 smithi117 ceph-mon[36002]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:26.186 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:25 smithi117 ceph-mon[36002]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:46:26.186 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:26.186 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:26.186 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:26.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:25 smithi120 ceph-mon[36826]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:26.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:25 smithi120 ceph-mon[36826]: osdmap e28: 4 total, 4 up, 4 in 2024-03-28T15:46:26.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:26.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:26.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:27.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:26 smithi117 ceph-mon[36002]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:27.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:26 smithi117 ceph-mon[36002]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:46:27.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:26 smithi120 ceph-mon[36826]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:27.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:26 smithi120 ceph-mon[36826]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:46:27.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:26 smithi023 ceph-mon[32448]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:27.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:26 smithi023 ceph-mon[32448]: osdmap e29: 4 total, 4 up, 4 in 2024-03-28T15:46:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:27 smithi023 ceph-mon[32448]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:28.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:27 smithi023 ceph-mon[32448]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:46:28.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:27 smithi117 ceph-mon[36002]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:28.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:27 smithi117 ceph-mon[36002]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:46:28.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:27 smithi120 ceph-mon[36826]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:28.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:27 smithi120 ceph-mon[36826]: osdmap e30: 4 total, 4 up, 4 in 2024-03-28T15:46:29.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/865796314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]': finished 2024-03-28T15:46:29.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T15:46:29.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:29.319 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:28 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/2342952635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/865796314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]': finished 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:29.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:28 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/2342952635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:29.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/865796314' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]: dispatch 2024-03-28T15:46:29.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "884d0555-eb5d-4206-8931-26bf05fe7e40"}]': finished 2024-03-28T15:46:29.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: osdmap e31: 5 total, 4 up, 5 in 2024-03-28T15:46:29.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:29.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:28 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/2342952635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:30.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:29 smithi117 ceph-mon[36002]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:30.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:29 smithi120 ceph-mon[36826]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:30.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:29 smithi023 ceph-mon[32448]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-28T15:46:32.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:31 smithi117 ceph-mon[36002]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-28T15:46:32.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:31 smithi120 ceph-mon[36826]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-28T15:46:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:31 smithi023 ceph-mon[32448]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-28T15:46:34.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:33 smithi117 ceph-mon[36002]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:34.338 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:33 smithi120 ceph-mon[36826]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:34.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:33 smithi023 ceph-mon[32448]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:36.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:35 smithi117 ceph-mon[36002]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:36.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:46:36.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:36.160 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:35 smithi117 ceph-mon[36002]: Deploying daemon osd.4 on smithi117 2024-03-28T15:46:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:35 smithi120 ceph-mon[36826]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:46:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:35 smithi120 ceph-mon[36826]: Deploying daemon osd.4 on smithi117 2024-03-28T15:46:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:35 smithi023 ceph-mon[32448]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-28T15:46:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:36.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:35 smithi023 ceph-mon[32448]: Deploying daemon osd.4 on smithi117 2024-03-28T15:46:38.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:37 smithi117 ceph-mon[36002]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:37 smithi120 ceph-mon[36826]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:38.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:37 smithi023 ceph-mon[32448]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:40.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:39 smithi117 ceph-mon[36002]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:39 smithi120 ceph-mon[36826]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:40.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:39 smithi023 ceph-mon[32448]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:42.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:41 smithi117 ceph-mon[36002]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:42.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:42.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:41 smithi120 ceph-mon[36826]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:42.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:41 smithi023 ceph-mon[32448]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:43.996 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:43 smithi117 ceph-mon[36002]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:44.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.067+0000 7f5f8e7fc700 1 -- 172.21.15.117:0/4226227313 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5f980625d0 con 0x7f5f840a2890 2024-03-28T15:46:44.068 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 4 on host 'smithi117' 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 msgr2=0x7f5f840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 0x7f5f840a4d50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009c10 tx=0x7f5f90009040 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 msgr2=0x7f5f98198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5f940098e0 tx=0x7f5f94009580 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 shutdown_connections 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5f840a2890 0x7f5f840a4d50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.069+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5f98103d60 0x7f5f9819fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.070+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f981021c0 0x7f5f98198480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.070+0000 7f5f9f701700 1 --2- 172.21.15.117:0/4226227313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5f981033c0 0x7f5f981989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.070+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 >> 172.21.15.117:0/4226227313 conn(0x7f5f980fd7a0 msgr2=0x7f5f98106790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.070+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 shutdown_connections 2024-03-28T15:46:44.070 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:46:44.070+0000 7f5f9f701700 1 -- 172.21.15.117:0/4226227313 wait complete. 2024-03-28T15:46:44.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:43 smithi120 ceph-mon[36826]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:43 smithi023 ceph-mon[32448]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:44.779 DEBUG:teuthology.orchestra.run.smithi117:osd.4> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.4.service 2024-03-28T15:46:44.782 INFO:tasks.cephadm:Deploying osd.5 on smithi120 with /dev/vg_nvme/lv_3... 2024-03-28T15:46:44.782 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-28T15:46:45.025 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:45.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:45 smithi120 ceph-mon[36826]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:45.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:45 smithi117 ceph-mon[36002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:45.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:45 smithi023 ceph-mon[32448]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-28T15:46:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:46:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:46:46.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:46.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-mon[36002]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.322 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T15:46:46.173+0000 7fb652652700 -1 osd.4 0 waiting for initial osdmap 2024-03-28T15:46:46.322 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 15:46:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T15:46:46.179+0000 7fb64e445700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.404 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:46 smithi120 ceph-mon[36826]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-28T15:46:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: osdmap e32: 5 total, 4 up, 5 in 2024-03-28T15:46:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:46.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: from='osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:46 smithi023 ceph-mon[32448]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:46:46.986 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-28T15:46:46.986 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-28T15:46:46.986 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-28T15:46:46.986 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-28T15:46:46.987 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111982 s, 936 MB/s 2024-03-28T15:46:46.987 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-28T15:46:47.442 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:47 smithi120 ceph-mon[36826]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:46:47.443 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:47 smithi120 ceph-mon[36826]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:46:47.443 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:47 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.443 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:47 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.505 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:47 smithi117 ceph-mon[36002]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:46:47.505 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:47 smithi117 ceph-mon[36002]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:46:47.505 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:47 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.505 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:47 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.627 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_3 2024-03-28T15:46:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:47 smithi023 ceph-mon[32448]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:46:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:47 smithi023 ceph-mon[32448]: osdmap e33: 5 total, 4 up, 5 in 2024-03-28T15:46:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:47 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:47 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:47.868 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] boot 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.243 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.244 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:48 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:46:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:46:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] boot 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:48.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:48 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:46:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:46:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-03-28T15:46:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: osd.4 [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] boot 2024-03-28T15:46:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: osdmap e34: 5 total, 5 up, 5 in 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:46:48.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:48 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:46:49.503 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:49 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi117 2024-03-28T15:46:49.503 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:49 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-28T15:46:49.504 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:49 smithi120 ceph-mon[36826]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:46:49.504 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 -- 172.21.15.120:0/2576340719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1022d0 msgr2=0x7f408c102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 --2- 172.21.15.120:0/2576340719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1022d0 0x7f408c102730 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4088009990 tx=0x7f4088009ca0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 -- 172.21.15.120:0/2576340719 shutdown_connections 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 --2- 172.21.15.120:0/2576340719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c102c70 0x7f408c109520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 --2- 172.21.15.120:0/2576340719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c1010d0 0x7f408c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 --2- 172.21.15.120:0/2576340719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1022d0 0x7f408c102730 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 -- 172.21.15.120:0/2576340719 >> 172.21.15.120:0/2576340719 conn(0x7f408c0fc650 msgr2=0x7f408c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 -- 172.21.15.120:0/2576340719 shutdown_connections 2024-03-28T15:46:49.505 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.504+0000 7f4093c7a700 1 -- 172.21.15.120:0/2576340719 wait complete. 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 Processor -- start 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 -- start start 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c1022d0 0x7f408c1036b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c102c70 0x7f408c103bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10cc90 con 0x7f408c1022d0 2024-03-28T15:46:49.507 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10cb10 con 0x7f408c102c70 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4093c7a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408c10ce10 con 0x7f408c1010d0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:40668/0 (socket says 172.21.15.120:40668) 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 -- 172.21.15.120:0/3763508089 learned_addr learned my addr 172.21.15.120:0/3763508089 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4092217700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c102c70 0x7f408c103bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091215700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c1022d0 0x7f408c1036b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 -- 172.21.15.120:0/3763508089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c102c70 msgr2=0x7f408c103bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c102c70 0x7f408c103bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 -- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c1022d0 msgr2=0x7f408c1036b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c1022d0 0x7f408c1036b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 -- 172.21.15.120:0/3763508089 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c1044a0 con 0x7f408c1010d0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4091a16700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c970 tx=0x7f407c00cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.505+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c004220 con 0x7f408c1010d0 2024-03-28T15:46:49.508 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.506+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f407c0043c0 con 0x7f408c1010d0 2024-03-28T15:46:49.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.506+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408c1a2550 con 0x7f408c1010d0 2024-03-28T15:46:49.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.506+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c003d40 con 0x7f408c1010d0 2024-03-28T15:46:49.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.506+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408c1a2a90 con 0x7f408c1010d0 2024-03-28T15:46:49.509 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.506+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f408c068e50 con 0x7f408c1010d0 2024-03-28T15:46:49.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.507+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f407c0076d0 con 0x7f408c1010d0 2024-03-28T15:46:49.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.508+0000 7f4082ffd700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 0x7f40780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:46:49.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.508+0000 7f4091215700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 0x7f40780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:46:49.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.508+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f407c0cd7c0 con 0x7f408c1010d0 2024-03-28T15:46:49.510 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.509+0000 7f4091215700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 0x7f40780a4dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f408800b5c0 tx=0x7f4088003a20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:46:49.513 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.513+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f407c014030 con 0x7f408c1010d0 2024-03-28T15:46:49.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:49 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi117 2024-03-28T15:46:49.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:49 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-28T15:46:49.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:49 smithi117 ceph-mon[36002]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:46:49.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:49 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi117 2024-03-28T15:46:49.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:49 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 8540M 2024-03-28T15:46:49.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:49 smithi023 ceph-mon[32448]: osdmap e35: 5 total, 5 up, 5 in 2024-03-28T15:46:49.718 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:46:49.718+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f408c0fe840 con 0x7f40780a2910 2024-03-28T15:46:50.506 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:50 smithi120 ceph-mon[36826]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:50.506 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:50.506 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:50.506 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:50 smithi117 ceph-mon[36002]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:50 smithi023 ceph-mon[32448]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:46:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:46:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:46:51.460 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:51 smithi120 ceph-mon[36826]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:51.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:51 smithi117 ceph-mon[36002]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:51 smithi023 ceph-mon[32448]: from='client.24229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:46:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:52 smithi117 ceph-mon[36002]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:52 smithi120 ceph-mon[36826]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:52 smithi023 ceph-mon[32448]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/3480677837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]': finished 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:46:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:53 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/1125117806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/3480677837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]': finished 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:46:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:53 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/1125117806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/3480677837' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]: dispatch 2024-03-28T15:46:53.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0743e053-1af5-44cd-9d42-efd7079b15f4"}]': finished 2024-03-28T15:46:53.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: osdmap e36: 6 total, 5 up, 6 in 2024-03-28T15:46:53.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:46:53.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:53 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1125117806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:46:54.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:54 smithi117 ceph-mon[36002]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:54 smithi120 ceph-mon[36826]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:54 smithi023 ceph-mon[32448]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:56.495 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:56 smithi120 ceph-mon[36826]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:56 smithi117 ceph-mon[36002]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:56 smithi023 ceph-mon[32448]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:58.405 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:46:58 smithi120 ceph-mon[36826]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:46:58 smithi117 ceph-mon[36002]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:46:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:46:58 smithi023 ceph-mon[32448]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:00 smithi117 ceph-mon[36002]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:47:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:00 smithi120 ceph-mon[36826]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:47:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:00 smithi023 ceph-mon[32448]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-28T15:47:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:01 smithi117 ceph-mon[36002]: Deploying daemon osd.5 on smithi120 2024-03-28T15:47:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:01 smithi120 ceph-mon[36826]: Deploying daemon osd.5 on smithi120 2024-03-28T15:47:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:01 smithi023 ceph-mon[32448]: Deploying daemon osd.5 on smithi120 2024-03-28T15:47:02.538 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:02 smithi120 ceph-mon[36826]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:02 smithi117 ceph-mon[36002]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:02 smithi023 ceph-mon[32448]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:03 smithi120 ceph-mon[36826]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:03 smithi023 ceph-mon[32448]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:03 smithi117 ceph-mon[36002]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:06.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:05 smithi117 ceph-mon[36002]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:06.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:06.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:06.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:06.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:05 smithi120 ceph-mon[36826]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:06.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:06.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:06.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:05 smithi023 ceph-mon[32448]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:06.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:08.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:07 smithi117 ceph-mon[36002]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:08.082 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:07 smithi120 ceph-mon[36826]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:08.133 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.133+0000 7f4082ffd700 1 -- 172.21.15.120:0/3763508089 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f408c0fe840 con 0x7f40780a2910 2024-03-28T15:47:08.133 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 5 on host 'smithi120' 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 msgr2=0x7f40780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 0x7f40780a4dd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f408800b5c0 tx=0x7f4088003a20 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 msgr2=0x7f408c105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f407c00c970 tx=0x7f407c00cc80 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 shutdown_connections 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40780a2910 0x7f40780a4dd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f408c102c70 0x7f408c103bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f408c1010d0 0x7f408c105060 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 --2- 172.21.15.120:0/3763508089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f408c1022d0 0x7f408c1036b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 >> 172.21.15.120:0/3763508089 conn(0x7f408c0fc650 msgr2=0x7f408c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:47:08.136 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 shutdown_connections 2024-03-28T15:47:08.137 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:47:08.135+0000 7f4093c7a700 1 -- 172.21.15.120:0/3763508089 wait complete. 2024-03-28T15:47:08.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:07 smithi023 ceph-mon[32448]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:08.745 DEBUG:teuthology.orchestra.run.smithi120:osd.5> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.5.service 2024-03-28T15:47:08.747 INFO:tasks.cephadm:Deploying osd.6 on smithi023 with /dev/vg_nvme/lv_2... 2024-03-28T15:47:08.747 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:47:09.002 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:47:09.393 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.394 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.486 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:10.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:10.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:47:10.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:47:10.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:10.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:10 smithi023 ceph-mon[32448]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.487 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-mon[36826]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:10.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-28T15:47:10.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: osdmap e37: 6 total, 5 up, 6 in 2024-03-28T15:47:10.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:10.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: from='osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:10 smithi117 ceph-mon[36002]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:47:10.782 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T15:47:10.682+0000 7fdb8ffe4700 -1 osd.5 0 waiting for initial osdmap 2024-03-28T15:47:10.782 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 15:47:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T15:47:10.690+0000 7fdb87dcf700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:47:11.033 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:47:11.033 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:47:11.033 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T15:47:11.033 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T15:47:11.033 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0160905 s, 652 MB/s 2024-03-28T15:47:11.034 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T15:47:11.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:11 smithi023 ceph-mon[32448]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:47:11.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:11 smithi023 ceph-mon[32448]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:47:11.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.344 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:11 smithi023 ceph-mon[32448]: from='osd.5 ' entity='osd.5' 2024-03-28T15:47:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:11 smithi117 ceph-mon[36002]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:47:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:11 smithi117 ceph-mon[36002]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:47:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:11 smithi117 ceph-mon[36002]: from='osd.5 ' entity='osd.5' 2024-03-28T15:47:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:11 smithi120 ceph-mon[36826]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:47:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:11 smithi120 ceph-mon[36826]: osdmap e38: 6 total, 5 up, 6 in 2024-03-28T15:47:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:11 smithi120 ceph-mon[36826]: from='osd.5 ' entity='osd.5' 2024-03-28T15:47:11.650 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_2 2024-03-28T15:47:11.898 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:47:12.279 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:47:12.279 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:47:12.280 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:12.280 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:12.280 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] boot 2024-03-28T15:47:12.280 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T15:47:12.280 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] boot 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T15:47:12.329 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:47:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:47:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-28T15:47:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: osd.5 [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] boot 2024-03-28T15:47:12.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: osdmap e39: 6 total, 6 up, 6 in 2024-03-28T15:47:12.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:47:13.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T15:47:13.274 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.275 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:13 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 -- 172.21.15.23:0/3691904835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0101160 msgr2=0x7f2ba0101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/3691904835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0101160 0x7f2ba0101580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2b900099c0 tx=0x7f2b90009cd0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 -- 172.21.15.23:0/3691904835 shutdown_connections 2024-03-28T15:47:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/3691904835 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0102d00 0x7f2ba01095b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/3691904835 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0101160 0x7f2ba0101580 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/3691904835 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba01027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.519+0000 7f2ba58e9700 1 -- 172.21.15.23:0/3691904835 >> 172.21.15.23:0/3691904835 conn(0x7f2ba00fc720 msgr2=0x7f2ba00feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- 172.21.15.23:0/3691904835 shutdown_connections 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- 172.21.15.23:0/3691904835 wait complete. 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 Processor -- start 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- start start 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0101160 0x7f2ba0198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba0198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:13.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010cd40 con 0x7f2ba0102d00 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010cbc0 con 0x7f2ba0101160 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.520+0000 7f2ba58e9700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba010cec0 con 0x7f2ba0102360 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55214/0 (socket says 172.21.15.23:55214) 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 -- 172.21.15.23:0/1782291280 learned_addr learned my addr 172.21.15.23:0/1782291280 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9effd700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0101160 0x7f2ba0198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:13.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9e7fc700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba0198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 -- 172.21.15.23:0/1782291280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 msgr2=0x7f2ba0198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba0198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 -- 172.21.15.23:0/1782291280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0101160 msgr2=0x7f2ba0198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0101160 0x7f2ba0198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 -- 172.21.15.23:0/1782291280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba01a00b0 con 0x7f2ba0102d00 2024-03-28T15:47:13.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9e7fc700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba0198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2b9f7fe700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800b9a0 tx=0x7f2b9800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98019070 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba01a03a0 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b9800ce10 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98007390 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.521+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba01a09b0 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.522+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba0068e50 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.523+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2b9801c070 con 0x7f2ba0102d00 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.524+0000 7f2ba48e7700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 0x7f2b880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:13.530 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.524+0000 7f2b9effd700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 0x7f2b880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.524+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f2b980ccf00 con 0x7f2ba0102d00 2024-03-28T15:47:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.524+0000 7f2b9effd700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 0x7f2b880a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000b580 tx=0x7f2b90005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:47:13.531 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.530+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2b9809ae40 con 0x7f2ba0102d00 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:13 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: osdmap e40: 6 total, 6 up, 6 in 2024-03-28T15:47:13.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.591 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:13.591 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:13.591 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:13.591 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:13 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:13.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:13.745+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ba01a0cd0 con 0x7f2b880a2890 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi120 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi120 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi120 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 8540M 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: osdmap e41: 6 total, 6 up, 6 in 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:15.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:15 smithi023 ceph-mon[32448]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:15.391 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:15 smithi023 ceph-mon[32448]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:47:15.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:15 smithi120 ceph-mon[36826]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:15.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:15 smithi120 ceph-mon[36826]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:47:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:15 smithi117 ceph-mon[36002]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:15 smithi117 ceph-mon[36002]: osdmap e42: 6 total, 6 up, 6 in 2024-03-28T15:47:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:16 smithi023 ceph-mon[32448]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:16 smithi023 ceph-mon[32448]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:47:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:16 smithi117 ceph-mon[36002]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:16 smithi117 ceph-mon[36002]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:47:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:16 smithi120 ceph-mon[36826]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 181 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:16 smithi120 ceph-mon[36826]: osdmap e43: 6 total, 6 up, 6 in 2024-03-28T15:47:17.472 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:17 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]: dispatch 2024-03-28T15:47:17.472 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:17 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]': finished 2024-03-28T15:47:17.472 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:17 smithi117 ceph-mon[36002]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T15:47:17.472 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:17.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:17 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]: dispatch 2024-03-28T15:47:17.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:17 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]': finished 2024-03-28T15:47:17.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:17 smithi120 ceph-mon[36826]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T15:47:17.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:17.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:17 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]: dispatch 2024-03-28T15:47:17.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:17 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2048244223' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "21218a07-2898-42e9-a82b-935c3bb50fcb"}]': finished 2024-03-28T15:47:17.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:17 smithi023 ceph-mon[32448]: osdmap e44: 7 total, 6 up, 7 in 2024-03-28T15:47:17.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:18 smithi117 ceph-mon[36002]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-28T15:47:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:18 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/972267763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:18 smithi120 ceph-mon[36826]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-28T15:47:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:18 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/972267763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:18 smithi023 ceph-mon[32448]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 138 KiB/s, 0 objects/s recovering 2024-03-28T15:47:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:18 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/972267763' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:20 smithi117 ceph-mon[36002]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-28T15:47:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:20 smithi120 ceph-mon[36826]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-28T15:47:20.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:20 smithi023 ceph-mon[32448]: pgmap v146: 1 pgs: 1 peering; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-28T15:47:22.390 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:22 smithi023 ceph-mon[32448]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T15:47:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:22 smithi117 ceph-mon[36002]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T15:47:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:22 smithi120 ceph-mon[36826]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-28T15:47:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:24 smithi117 ceph-mon[36002]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:47:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:24 smithi120 ceph-mon[36826]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:47:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:24 smithi023 ceph-mon[32448]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-28T15:47:25.320 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:47:25.320 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:47:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-28T15:47:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:26.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:26 smithi023 ceph-mon[32448]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:26.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:26 smithi023 ceph-mon[32448]: Deploying daemon osd.6 on smithi023 2024-03-28T15:47:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:26 smithi117 ceph-mon[36002]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:26 smithi117 ceph-mon[36002]: Deploying daemon osd.6 on smithi023 2024-03-28T15:47:26.588 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:26 smithi120 ceph-mon[36826]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:26 smithi120 ceph-mon[36826]: Deploying daemon osd.6 on smithi023 2024-03-28T15:47:28.411 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:28 smithi023 ceph-mon[32448]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:28 smithi117 ceph-mon[36002]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:28 smithi120 ceph-mon[36826]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:30.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:30 smithi023 ceph-mon[32448]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:30.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.372 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:30 smithi117 ceph-mon[36002]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:30 smithi120 ceph-mon[36826]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:32.524 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:32 smithi023 ceph-mon[32448]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:32 smithi117 ceph-mon[36002]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:32 smithi120 ceph-mon[36826]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:33.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.852+0000 7f2ba48e7700 1 -- 172.21.15.23:0/1782291280 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2ba01a0cd0 con 0x7f2b880a2890 2024-03-28T15:47:33.852 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 6 on host 'smithi023' 2024-03-28T15:47:33.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 msgr2=0x7f2b880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 0x7f2b880a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2b9000b580 tx=0x7f2b90005fb0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 msgr2=0x7f2ba019fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800b9a0 tx=0x7f2b9800bcb0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 shutdown_connections 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2b880a2890 0x7f2b880a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2ba0102d00 0x7f2ba019fa10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ba0101160 0x7f2ba0198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 --2- 172.21.15.23:0/1782291280 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2ba0102360 0x7f2ba0198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 >> 172.21.15.23:0/1782291280 conn(0x7f2ba00fc720 msgr2=0x7f2ba00fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 shutdown_connections 2024-03-28T15:47:33.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:47:33.854+0000 7f2ba58e9700 1 -- 172.21.15.23:0/1782291280 wait complete. 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:34.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:33 smithi023 ceph-mon[32448]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:34.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:33 smithi117 ceph-mon[36002]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:47:34.342 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:34.343 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:33 smithi120 ceph-mon[36826]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-28T15:47:34.572 DEBUG:teuthology.orchestra.run.smithi023:osd.6> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.6.service 2024-03-28T15:47:34.574 INFO:tasks.cephadm:Deploying osd.7 on smithi117 with /dev/vg_nvme/lv_2... 2024-03-28T15:47:34.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:47:34.823 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:47:36.110 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-mon[32448]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:36.110 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-mon[32448]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:47:36.110 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-mon[32448]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T15:47:36.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-mon[32448]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:47:36.111 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:36.111 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T15:47:35.852+0000 7effc7978700 -1 osd.6 0 waiting for initial osdmap 2024-03-28T15:47:36.111 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 15:47:35 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T15:47:35.862+0000 7effc376b700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:47:36.116 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:35 smithi117 ceph-mon[36002]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:36.116 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:35 smithi117 ceph-mon[36002]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:47:36.116 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:35 smithi117 ceph-mon[36002]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T15:47:36.116 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:35 smithi117 ceph-mon[36002]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:47:36.116 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:35 smithi120 ceph-mon[36826]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:35 smithi120 ceph-mon[36826]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-28T15:47:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:35 smithi120 ceph-mon[36826]: osdmap e45: 7 total, 6 up, 7 in 2024-03-28T15:47:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:35 smithi120 ceph-mon[36826]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:47:36.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:36.757 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113341 s, 925 MB/s 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 2024-03-28T15:47:36.758 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: from='osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: osdmap e46: 7 total, 6 up, 7 in 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:37.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-03-28T15:47:37.704 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:47:38.025 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:38.025 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] boot 2024-03-28T15:47:38.025 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T15:47:38.025 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:38.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:37 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] boot 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:38.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:37 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 185 MiB used, 536 GiB / 536 GiB avail 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: osd.6 [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] boot 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: osdmap e47: 7 total, 7 up, 7 in 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:47:38.340 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:37 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:39.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:38 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi023 2024-03-28T15:47:39.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:38 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T15:47:39.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:38 smithi117 ceph-mon[36002]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T15:47:39.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:38 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi023 2024-03-28T15:47:39.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:38 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T15:47:39.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:38 smithi023 ceph-mon[32448]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T15:47:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:38 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi023 2024-03-28T15:47:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:38 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 5696M 2024-03-28T15:47:39.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:38 smithi120 ceph-mon[36826]: osdmap e48: 7 total, 7 up, 7 in 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.351+0000 7f4316f34700 1 -- 172.21.15.117:0/1688118375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 msgr2=0x7f43101014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.351+0000 7f4316f34700 1 --2- 172.21.15.117:0/1688118375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f43101014e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43040099c0 tx=0x7f4304009cd0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.351+0000 7f4316f34700 1 -- 172.21.15.117:0/1688118375 shutdown_connections 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.351+0000 7f4316f34700 1 --2- 172.21.15.117:0/1688118375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4310102c60 0x7f4310109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.351+0000 7f4316f34700 1 --2- 172.21.15.117:0/1688118375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f43101014e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 --2- 172.21.15.117:0/1688118375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43101022c0 0x7f4310102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- 172.21.15.117:0/1688118375 >> 172.21.15.117:0/1688118375 conn(0x7f43100fc680 msgr2=0x7f43100feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:47:39.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- 172.21.15.117:0/1688118375 shutdown_connections 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- 172.21.15.117:0/1688118375 wait complete. 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 Processor -- start 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- start start 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43101022c0 0x7f431019f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4310102c60 0x7f43101a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431010cd00 con 0x7f4310102c60 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431010cb80 con 0x7f43101010c0 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4316f34700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431010ce80 con 0x7f43101022c0 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4314cd0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4314cd0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:54424/0 (socket says 172.21.15.117:54424) 2024-03-28T15:47:39.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.352+0000 7f4314cd0700 1 -- 172.21.15.117:0/143358316 learned_addr learned my addr 172.21.15.117:0/143358316 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f430ffff700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43101022c0 0x7f431019f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f43154d1700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4310102c60 0x7f43101a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 -- 172.21.15.117:0/143358316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43101022c0 msgr2=0x7f431019f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43101022c0 0x7f431019f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 -- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4310102c60 msgr2=0x7f43101a1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4310102c60 0x7f43101a1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 -- 172.21.15.117:0/143358316 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43101a2240 con 0x7f43101010c0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4314cd0700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f430400c010 tx=0x7f4304005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304027070 con 0x7f43101010c0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43040041d0 con 0x7f43101010c0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4304022e60 con 0x7f43101010c0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43101a24d0 con 0x7f43101010c0 2024-03-28T15:47:39.356 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.353+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43101a2a80 con 0x7f43101010c0 2024-03-28T15:47:39.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.354+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4310068e50 con 0x7f43101010c0 2024-03-28T15:47:39.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.355+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f430402a070 con 0x7f43101010c0 2024-03-28T15:47:39.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.356+0000 7f430dffb700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 0x7f43000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:47:39.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.356+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f43040cf230 con 0x7f43101010c0 2024-03-28T15:47:39.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.356+0000 7f430ffff700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 0x7f43000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:47:39.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.356+0000 7f430ffff700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 0x7f43000a4dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006fd0 tx=0x7f42fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:47:39.359 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.359+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4304013070 con 0x7f43101010c0 2024-03-28T15:47:39.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:39.563+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f43100625d0 con 0x7f43000a2910 2024-03-28T15:47:40.170 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:39 smithi117 ceph-mon[36002]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:40.170 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:40.170 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:40.170 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:39 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:40.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:39 smithi023 ceph-mon[32448]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:40.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:40.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:40.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:39 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:39 smithi120 ceph-mon[36826]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:47:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:47:40.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:39 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:41.164 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:40 smithi117 ceph-mon[36002]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:41.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:40 smithi023 ceph-mon[32448]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:41.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:40 smithi120 ceph-mon[36826]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:47:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:41 smithi023 ceph-mon[32448]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:41 smithi117 ceph-mon[36002]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:42.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:41 smithi120 ceph-mon[36826]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:42 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/3399226730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:42 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:42 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]': finished 2024-03-28T15:47:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:42 smithi117 ceph-mon[36002]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T15:47:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:42 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:47:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:42 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/3399226730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:42 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:42 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]': finished 2024-03-28T15:47:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:42 smithi023 ceph-mon[32448]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T15:47:43.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:42 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:47:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:42 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/3399226730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:42 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]: dispatch 2024-03-28T15:47:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:42 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "18720d2e-c5ab-4b3e-b8ee-8b2b687daabf"}]': finished 2024-03-28T15:47:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:42 smithi120 ceph-mon[36826]: osdmap e49: 8 total, 7 up, 8 in 2024-03-28T15:47:43.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:42 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:47:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:43 smithi023 ceph-mon[32448]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:44.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:43 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/2652059983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:44.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:43 smithi117 ceph-mon[36002]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:44.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:43 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/2652059983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:44.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:43 smithi120 ceph-mon[36826]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:44.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:43 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/2652059983' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:47:46.141 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:45 smithi120 ceph-mon[36826]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:46.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:45 smithi023 ceph-mon[32448]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:46.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:45 smithi117 ceph-mon[36002]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:48.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:47 smithi117 ceph-mon[36002]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:48.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:47 smithi023 ceph-mon[32448]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:48.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:47 smithi120 ceph-mon[36826]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:50.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:49 smithi117 ceph-mon[36002]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:50.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:49 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:47:50.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:49 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:50.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:49 smithi023 ceph-mon[32448]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:50.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:49 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:47:50.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:49 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:50.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:49 smithi120 ceph-mon[36826]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:50.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:49 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-28T15:47:50.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:49 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:47:51.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:50 smithi117 ceph-mon[36002]: Deploying daemon osd.7 on smithi117 2024-03-28T15:47:51.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:50 smithi023 ceph-mon[32448]: Deploying daemon osd.7 on smithi117 2024-03-28T15:47:51.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:50 smithi120 ceph-mon[36826]: Deploying daemon osd.7 on smithi117 2024-03-28T15:47:52.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:51 smithi117 ceph-mon[36002]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:52.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:51 smithi023 ceph-mon[32448]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:52.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:51 smithi120 ceph-mon[36826]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:54.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:53 smithi023 ceph-mon[32448]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:54.241 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:53 smithi117 ceph-mon[36002]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:54.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:53 smithi120 ceph-mon[36826]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:56.048 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:55 smithi117 ceph-mon[36002]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:56.048 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:56.048 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:56.048 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:56.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:55 smithi023 ceph-mon[32448]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:56.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:56.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:56.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:56.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:55 smithi120 ceph-mon[36826]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:56.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:47:56.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:56.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:58.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:57 smithi023 ceph-mon[32448]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:58.177 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:57 smithi117 ceph-mon[36002]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:58.234 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 7 on host 'smithi117' 2024-03-28T15:47:58.235 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.232+0000 7f430dffb700 1 -- 172.21.15.117:0/143358316 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f43100625d0 con 0x7f43000a2910 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 msgr2=0x7f43000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 0x7f43000a4dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f42fc006fd0 tx=0x7f42fc008040 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 msgr2=0x7f431006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f430400c010 tx=0x7f4304005e80 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 shutdown_connections 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f43000a2910 0x7f43000a4dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4310102c60 0x7f43101a1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43101010c0 0x7f431006b410 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 --2- 172.21.15.117:0/143358316 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43101022c0 0x7f431019f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 >> 172.21.15.117:0/143358316 conn(0x7f43100fc680 msgr2=0x7f43101055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:47:58.236 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 shutdown_connections 2024-03-28T15:47:58.237 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:47:58.234+0000 7f4316f34700 1 -- 172.21.15.117:0/143358316 wait complete. 2024-03-28T15:47:58.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:57 smithi120 ceph-mon[36826]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:47:59.200 DEBUG:teuthology.orchestra.run.smithi117:osd.7> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.7.service 2024-03-28T15:47:59.203 INFO:tasks.cephadm:Deploying osd.8 on smithi120 with /dev/vg_nvme/lv_2... 2024-03-28T15:47:59.203 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-28T15:47:59.450 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:47:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:59 smithi117 ceph-mon[36002]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:47:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:47:59 smithi117 ceph-mon[36002]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:47:59.572 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.572 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.572 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:59 smithi120 ceph-mon[36826]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:47:59.572 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:47:59 smithi120 ceph-mon[36826]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:47:59.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:47:59.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:59 smithi023 ceph-mon[32448]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:47:59.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:47:59 smithi023 ceph-mon[32448]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:00.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:00.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: osdmap e50: 8 total, 7 up, 8 in 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:48:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:01.031 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T15:48:00.639+0000 7f735f66b700 -1 osd.7 0 waiting for initial osdmap 2024-03-28T15:48:01.031 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 15:48:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T15:48:00.647+0000 7f7357456700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:48:01.294 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:48:01.294 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:48:01.294 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.294 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: from='osd.7 ' entity='osd.7' 2024-03-28T15:48:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:01 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: from='osd.7 ' entity='osd.7' 2024-03-28T15:48:01.507 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:01 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.614 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-28T15:48:01.615 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-28T15:48:01.615 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-28T15:48:01.615 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-28T15:48:01.615 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113803 s, 921 MB/s 2024-03-28T15:48:01.615 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: osdmap e51: 8 total, 7 up, 8 in 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: from='osd.7 ' entity='osd.7' 2024-03-28T15:48:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:01 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:02.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_2 2024-03-28T15:48:02.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:48:02.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:48:02.456 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:02.457 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] boot 2024-03-28T15:48:02.457 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T15:48:02.457 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:02 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:02.471 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] boot 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T15:48:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:02 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:48:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:48:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 212 MiB used, 626 GiB / 626 GiB avail 2024-03-28T15:48:02.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: osd.7 [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] boot 2024-03-28T15:48:02.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: osdmap e52: 8 total, 8 up, 8 in 2024-03-28T15:48:02.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:02 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:48:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T15:48:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:03.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: osdmap e53: 8 total, 8 up, 8 in 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:03.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:03.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:03.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 -- 172.21.15.120:0/3888264713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141022c0 msgr2=0x7f3414102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 --2- 172.21.15.120:0/3888264713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141022c0 0x7f3414102720 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3410009990 tx=0x7f3410009ca0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 -- 172.21.15.120:0/3888264713 shutdown_connections 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 --2- 172.21.15.120:0/3888264713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3414102c60 0x7f3414109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 --2- 172.21.15.120:0/3888264713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34141010c0 0x7f34141014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 --2- 172.21.15.120:0/3888264713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141022c0 0x7f3414102720 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 -- 172.21.15.120:0/3888264713 >> 172.21.15.120:0/3888264713 conn(0x7f34140fc680 msgr2=0x7f34140feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 -- 172.21.15.120:0/3888264713 shutdown_connections 2024-03-28T15:48:04.193 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.190+0000 7f341c6a8700 1 -- 172.21.15.120:0/3888264713 wait complete. 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 Processor -- start 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 -- start start 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34141022c0 0x7f341410a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3414102c60 0x7f341410a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341410dea0 con 0x7f34141022c0 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341410dd20 con 0x7f3414102c60 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341c6a8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341410e020 con 0x7f34141010c0 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:44764/0 (socket says 172.21.15.120:44764) 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 -- 172.21.15.120:0/335562029 learned_addr learned my addr 172.21.15.120:0/335562029 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:48:04.194 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341ac45700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3414102c60 0x7f341410a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f3419c43700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34141022c0 0x7f341410a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 -- 172.21.15.120:0/335562029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3414102c60 msgr2=0x7f341410a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3414102c60 0x7f341410a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 -- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34141022c0 msgr2=0x7f341410a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.191+0000 7f341a444700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34141022c0 0x7f341410a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f341a444700 1 -- 172.21.15.120:0/335562029 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341410aed0 con 0x7f34141010c0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f341a444700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f340400b500 tx=0x7f340400b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3404015020 con 0x7f34141010c0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34141a26a0 con 0x7f34141010c0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3404011d10 con 0x7f34141010c0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f340400f2e0 con 0x7f34141010c0 2024-03-28T15:48:04.195 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.192+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34141a2c60 con 0x7f34141010c0 2024-03-28T15:48:04.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.193+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3414068e50 con 0x7f34141010c0 2024-03-28T15:48:04.196 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.194+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3404018020 con 0x7f34141010c0 2024-03-28T15:48:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.194+0000 7f340b7fe700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 0x7f34000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.195+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5397+0+0 (secure 0 0 0) 0x7f34040cdbb0 con 0x7f34141010c0 2024-03-28T15:48:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.195+0000 7f3419c43700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 0x7f34000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:04.197 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.195+0000 7f3419c43700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 0x7f34000a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f341000b5c0 tx=0x7f3410005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:04.198 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.198+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f340401f020 con 0x7f34141010c0 2024-03-28T15:48:04.407 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:04.406+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f34140625d0 con 0x7f34000a2890 2024-03-28T15:48:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:04 smithi117 ceph-mon[36002]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:04 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi117 2024-03-28T15:48:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:04 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-28T15:48:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:04 smithi117 ceph-mon[36002]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T15:48:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:04 smithi117 ceph-mon[36002]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:48:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:04 smithi120 ceph-mon[36826]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:04 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi117 2024-03-28T15:48:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:04 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-28T15:48:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:04 smithi120 ceph-mon[36826]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T15:48:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:04 smithi120 ceph-mon[36826]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:48:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:04 smithi023 ceph-mon[32448]: pgmap v177: 1 pgs: 1 remapped; 577 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:04 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi117 2024-03-28T15:48:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:04 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 5693M 2024-03-28T15:48:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:04 smithi023 ceph-mon[32448]: osdmap e54: 8 total, 8 up, 8 in 2024-03-28T15:48:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:04 smithi023 ceph-mon[32448]: osdmap e55: 8 total, 8 up, 8 in 2024-03-28T15:48:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:05 smithi117 ceph-mon[36002]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:48:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:05 smithi120 ceph-mon[36826]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:48:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:05 smithi023 ceph-mon[32448]: osdmap e56: 8 total, 8 up, 8 in 2024-03-28T15:48:06.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:06 smithi120 ceph-mon[36826]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:06.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:06 smithi120 ceph-mon[36826]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:06.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:06 smithi117 ceph-mon[36002]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:06.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:06 smithi117 ceph-mon[36002]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:06 smithi023 ceph-mon[32448]: pgmap v180: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:06 smithi023 ceph-mon[32448]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:07.556 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:07 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/750395656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.557 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:07 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.557 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:07 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]': finished 2024-03-28T15:48:07.557 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:07 smithi120 ceph-mon[36826]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T15:48:07.557 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:07 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/750395656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:07 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:07 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]': finished 2024-03-28T15:48:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:07 smithi117 ceph-mon[36002]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T15:48:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:07 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/750395656' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:07 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]: dispatch 2024-03-28T15:48:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:07 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "692e7ca1-8a89-4b1d-a931-f7e9bb788110"}]': finished 2024-03-28T15:48:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:07 smithi023 ceph-mon[32448]: osdmap e57: 9 total, 8 up, 9 in 2024-03-28T15:48:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:08 smithi117 ceph-mon[36002]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:08 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/1197500755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:08 smithi120 ceph-mon[36826]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:08 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/1197500755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:08 smithi023 ceph-mon[32448]: pgmap v182: 1 pgs: 1 remapped; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:08 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1197500755' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:10.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:10 smithi117 ceph-mon[36002]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:10.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:10 smithi120 ceph-mon[36826]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:10 smithi023 ceph-mon[32448]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:12 smithi117 ceph-mon[36002]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:12.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:12 smithi120 ceph-mon[36826]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:12 smithi023 ceph-mon[32448]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:14 smithi117 ceph-mon[36002]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:14 smithi120 ceph-mon[36826]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:14 smithi023 ceph-mon[32448]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:15.497 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:48:15.497 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:48:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:15.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-28T15:48:15.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:16 smithi117 ceph-mon[36002]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:16 smithi117 ceph-mon[36002]: Deploying daemon osd.8 on smithi120 2024-03-28T15:48:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:16 smithi120 ceph-mon[36826]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:16 smithi120 ceph-mon[36826]: Deploying daemon osd.8 on smithi120 2024-03-28T15:48:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:16 smithi023 ceph-mon[32448]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:16 smithi023 ceph-mon[32448]: Deploying daemon osd.8 on smithi120 2024-03-28T15:48:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:18 smithi117 ceph-mon[36002]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:18.577 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:18 smithi120 ceph-mon[36826]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:18 smithi023 ceph-mon[32448]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:20 smithi117 ceph-mon[36002]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:20 smithi120 ceph-mon[36826]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:20 smithi023 ceph-mon[32448]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:21 smithi023 ceph-mon[32448]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:21 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:21 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:21 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:21.780 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:21 smithi120 ceph-mon[36826]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:21.780 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:21 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.780 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:21 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.780 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:21 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:21 smithi117 ceph-mon[36002]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:21 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:21 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:21 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:23.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:23.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:23.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:23.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.372 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.371+0000 7f340b7fe700 1 -- 172.21.15.120:0/335562029 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f34140625d0 con 0x7f34000a2890 2024-03-28T15:48:24.372 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 8 on host 'smithi120' 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 msgr2=0x7f34000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 0x7f34000a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f341000b5c0 tx=0x7f3410005fb0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 msgr2=0x7f341410ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f340400b500 tx=0x7f340400b8c0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 shutdown_connections 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f34000a2890 0x7f34000a4d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3414102c60 0x7f341410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.375 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34141010c0 0x7f341410ba90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 --2- 172.21.15.120:0/335562029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34141022c0 0x7f341410a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:24.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 >> 172.21.15.120:0/335562029 conn(0x7f34140fc680 msgr2=0x7f3414105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:48:24.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 shutdown_connections 2024-03-28T15:48:24.376 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:48:24.374+0000 7f341c6a8700 1 -- 172.21.15.120:0/335562029 wait complete. 2024-03-28T15:48:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:24 smithi120 ceph-mon[36826]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:24 smithi120 ceph-mon[36826]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:24 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:24 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:24 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:24 smithi023 ceph-mon[32448]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:24 smithi023 ceph-mon[32448]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:24 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:24 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:24 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:24 smithi117 ceph-mon[36002]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:24 smithi117 ceph-mon[36002]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-28T15:48:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:24 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:24 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:24 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:25.008 DEBUG:teuthology.orchestra.run.smithi120:osd.8> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.8.service 2024-03-28T15:48:25.011 INFO:tasks.cephadm:Deploying osd.9 on smithi023 with /dev/vg_nvme/lv_1... 2024-03-28T15:48:25.011 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:48:25.265 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-mon[36826]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:25.590 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T15:48:25.434+0000 7fb6717dc700 -1 osd.8 0 waiting for initial osdmap 2024-03-28T15:48:25.590 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 15:48:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T15:48:25.442+0000 7fb66d5cf700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:48:25.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:25.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:48:25.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T15:48:25.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:25.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:25.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:25 smithi023 ceph-mon[32448]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: osdmap e58: 9 total, 8 up, 9 in 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: from='osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:25 smithi117 ceph-mon[36002]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:48:26.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:26 smithi023 ceph-mon[32448]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:48:26.661 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:26 smithi023 ceph-mon[32448]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T15:48:26.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:26.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:26 smithi117 ceph-mon[36002]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:48:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:26 smithi117 ceph-mon[36002]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T15:48:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:26 smithi120 ceph-mon[36826]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:48:26.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:26 smithi120 ceph-mon[36826]: osdmap e59: 9 total, 8 up, 9 in 2024-03-28T15:48:26.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:26.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr: stderr: 10+0 records in 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:10+0 records out 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0161871 s, 648 MB/s 2024-03-28T15:48:27.269 INFO:teuthology.orchestra.run.smithi023.stderr:--> Zapping successful for: 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] boot 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T15:48:27.567 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:27 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] boot 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T15:48:27.584 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:27 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 715 GiB / 715 GiB avail 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: osd.8 [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] boot 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: osdmap e60: 9 total, 9 up, 9 in 2024-03-28T15:48:27.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:27 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:48:27.996 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi023:vg_nvme/lv_1 2024-03-28T15:48:28.242 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi120 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-28T15:48:28.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:28.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:28.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi120 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:28.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: osdmap e61: 9 total, 9 up, 9 in 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi120 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 5693M 2024-03-28T15:48:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:28.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:28.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:28.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:29.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:29 smithi023 ceph-mon[32448]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:29.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:29 smithi023 ceph-mon[32448]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T15:48:29.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:29 smithi023 ceph-mon[32448]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:48:29.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:29 smithi117 ceph-mon[36002]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:29.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:29 smithi117 ceph-mon[36002]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T15:48:29.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:29 smithi117 ceph-mon[36002]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:48:29.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:29 smithi120 ceph-mon[36826]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:29.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:29 smithi120 ceph-mon[36826]: osdmap e62: 9 total, 9 up, 9 in 2024-03-28T15:48:29.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:29 smithi120 ceph-mon[36826]: osdmap e63: 9 total, 9 up, 9 in 2024-03-28T15:48:29.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 -- 172.21.15.23:0/3733473060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 msgr2=0x7f0f0c10a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:29.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 --2- 172.21.15.23:0/3733473060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c10a320 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0f04009990 tx=0x7f0f04009ca0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 -- 172.21.15.23:0/3733473060 shutdown_connections 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 --2- 172.21.15.23:0/3733473060 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c10a320 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 --2- 172.21.15.23:0/3733473060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c102cb0 0x7f0f0c1050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 --2- 172.21.15.23:0/3733473060 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c1079d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 -- 172.21.15.23:0/3733473060 >> 172.21.15.23:0/3733473060 conn(0x7f0f0c0fc660 msgr2=0x7f0f0c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 -- 172.21.15.23:0/3733473060 shutdown_connections 2024-03-28T15:48:29.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.934+0000 7f0f10f92700 1 -- 172.21.15.23:0/3733473060 wait complete. 2024-03-28T15:48:29.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 Processor -- start 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 -- start start 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c102cb0 0x7f0f0c19a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c19aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c10ddf0 con 0x7f0f0c107f10 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c10dc70 con 0x7f0f0c1055e0 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f10f92700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c10df70 con 0x7f0f0c102cb0 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48312/0 (socket says 172.21.15.23:48312) 2024-03-28T15:48:29.939 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 -- 172.21.15.23:0/1861481785 learned_addr learned my addr 172.21.15.23:0/1861481785 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f09d9b700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c19aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0a59c700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c102cb0 0x7f0f0c19a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 -- 172.21.15.23:0/1861481785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c102cb0 msgr2=0x7f0f0c19a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c102cb0 0x7f0f0c19a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 -- 172.21.15.23:0/1861481785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 msgr2=0x7f0f0c19aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c19aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 -- 172.21.15.23:0/1861481785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c1a21f0 con 0x7f0f0c107f10 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f09d9b700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.935+0000 7f0f0ad9d700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f04006010 tx=0x7f0f0400bc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.936+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04026070 con 0x7f0f0c107f10 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.936+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c1a2480 con 0x7f0f0c107f10 2024-03-28T15:48:29.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.936+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f0400be40 con 0x7f0f0c107f10 2024-03-28T15:48:29.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.936+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f04021700 con 0x7f0f0c107f10 2024-03-28T15:48:29.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.936+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c1a2930 con 0x7f0f0c107f10 2024-03-28T15:48:29.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.937+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c068e50 con 0x7f0f0c107f10 2024-03-28T15:48:29.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.938+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0f04029070 con 0x7f0f0c107f10 2024-03-28T15:48:29.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.938+0000 7f0f037fe700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 0x7f0ef80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:29.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.938+0000 7f0f0a59c700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 0x7f0ef80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:29.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.938+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f0f040cf0c0 con 0x7f0f0c107f10 2024-03-28T15:48:29.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.939+0000 7f0f0a59c700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 0x7f0ef80a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009380 tx=0x7f0efc00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:29.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:29.943+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0f0405d740 con 0x7f0f0c107f10 2024-03-28T15:48:30.161 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:30.159+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f0c0625d0 con 0x7f0ef80a2890 2024-03-28T15:48:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:30 smithi023 ceph-mon[32448]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:30.779 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:30 smithi023 ceph-mon[32448]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:48:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:30 smithi117 ceph-mon[36002]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:30 smithi117 ceph-mon[36002]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:48:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:30 smithi120 ceph-mon[36826]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi023:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:30 smithi120 ceph-mon[36826]: osdmap e64: 9 total, 9 up, 9 in 2024-03-28T15:48:32.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:32 smithi023 ceph-mon[32448]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:32 smithi117 ceph-mon[36002]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:32 smithi120 ceph-mon[36826]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:33.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:33 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]: dispatch 2024-03-28T15:48:33.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:33 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]': finished 2024-03-28T15:48:33.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:33 smithi117 ceph-mon[36002]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T15:48:33.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:33.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:33 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]: dispatch 2024-03-28T15:48:33.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:33 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]': finished 2024-03-28T15:48:33.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:33 smithi120 ceph-mon[36826]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T15:48:33.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:33 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]: dispatch 2024-03-28T15:48:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:33 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1007457364' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4"}]': finished 2024-03-28T15:48:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:33 smithi023 ceph-mon[32448]: osdmap e65: 10 total, 9 up, 10 in 2024-03-28T15:48:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:34 smithi117 ceph-mon[36002]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-28T15:48:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:34 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1042688388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:34 smithi120 ceph-mon[36826]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-28T15:48:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:34 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1042688388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:34 smithi023 ceph-mon[32448]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-28T15:48:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:34 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1042688388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:48:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:36 smithi117 ceph-mon[36002]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T15:48:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:36 smithi120 ceph-mon[36826]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T15:48:36.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:36 smithi023 ceph-mon[32448]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-28T15:48:38.353 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:38 smithi023 ceph-mon[32448]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T15:48:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:38 smithi117 ceph-mon[36002]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T15:48:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:38 smithi120 ceph-mon[36826]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-03-28T15:48:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:40 smithi117 ceph-mon[36002]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:48:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:40 smithi120 ceph-mon[36826]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:48:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:40 smithi023 ceph-mon[32448]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:48:41.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:48:41.536 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:41.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:48:41.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:41.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-28T15:48:41.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:42.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:42 smithi117 ceph-mon[36002]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:48:42.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:42 smithi117 ceph-mon[36002]: Deploying daemon osd.9 on smithi023 2024-03-28T15:48:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:42 smithi120 ceph-mon[36826]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:48:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:42 smithi120 ceph-mon[36826]: Deploying daemon osd.9 on smithi023 2024-03-28T15:48:42.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:42 smithi023 ceph-mon[32448]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-28T15:48:42.642 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:42 smithi023 ceph-mon[32448]: Deploying daemon osd.9 on smithi023 2024-03-28T15:48:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:44 smithi117 ceph-mon[36002]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:44.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:44 smithi023 ceph-mon[32448]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:44 smithi120 ceph-mon[36826]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:46.242 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:46 smithi023 ceph-mon[32448]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:46 smithi117 ceph-mon[36002]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:46 smithi120 ceph-mon[36826]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:47.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:47 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:47.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:47 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:47.314 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:47 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:47.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:47 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:47.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:47 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:47.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:47 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:47.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:47 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:47.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:47 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:47.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:47 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:48 smithi117 ceph-mon[36002]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:48.583 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:48 smithi120 ceph-mon[36826]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:48.628 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:48 smithi023 ceph-mon[32448]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:50.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:50.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.542 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:50.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:50.543 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:50 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:50 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:50 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:50.719 INFO:teuthology.orchestra.run.smithi023.stdout:Created osd(s) 9 on host 'smithi023' 2024-03-28T15:48:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.718+0000 7f0f037fe700 1 -- 172.21.15.23:0/1861481785 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f0c0625d0 con 0x7f0ef80a2890 2024-03-28T15:48:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 msgr2=0x7f0ef80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 0x7f0ef80a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0efc009380 tx=0x7f0efc00b040 comp rx=0 tx=0).stop 2024-03-28T15:48:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 msgr2=0x7f0f0c1a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0f04006010 tx=0x7f0f0400bc60 comp rx=0 tx=0).stop 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 shutdown_connections 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0ef80a2890 0x7f0ef80a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0f0c107f10 0x7f0f0c1a1b50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0f0c102cb0 0x7f0f0c19a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 --2- 172.21.15.23:0/1861481785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f0c1055e0 0x7f0f0c19aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 >> 172.21.15.23:0/1861481785 conn(0x7f0f0c0fc660 msgr2=0x7f0f0c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 shutdown_connections 2024-03-28T15:48:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:48:50.720+0000 7f0f10f92700 1 -- 172.21.15.23:0/1861481785 wait complete. 2024-03-28T15:48:51.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:51 smithi023 ceph-mon[32448]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:48:51.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:51 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:51.459 DEBUG:teuthology.orchestra.run.smithi023:osd.9> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.9.service 2024-03-28T15:48:51.461 INFO:tasks.cephadm:Deploying osd.10 on smithi117 with /dev/vg_nvme/lv_1... 2024-03-28T15:48:51.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:48:51.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:51 smithi117 ceph-mon[36002]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:48:51.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:51 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:51.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:51 smithi120 ceph-mon[36826]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-28T15:48:51.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:51.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:51 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:48:51.710 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:48:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:52 smithi117 ceph-mon[36002]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:52 smithi117 ceph-mon[36002]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:48:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:52 smithi117 ceph-mon[36002]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T15:48:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:52 smithi117 ceph-mon[36002]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:48:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:52 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:52 smithi120 ceph-mon[36826]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:52 smithi120 ceph-mon[36826]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:48:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:52 smithi120 ceph-mon[36826]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T15:48:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:52 smithi120 ceph-mon[36826]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:48:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:52 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:52.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-mon[32448]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:52.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-mon[32448]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-28T15:48:52.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-mon[32448]: osdmap e66: 10 total, 9 up, 10 in 2024-03-28T15:48:52.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-mon[32448]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]: dispatch 2024-03-28T15:48:52.621 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:52.622 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T15:48:52.285+0000 7fc378a16700 -1 osd.9 0 waiting for initial osdmap 2024-03-28T15:48:52.622 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 15:48:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T15:48:52.297+0000 7fc374809700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:48:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:53 smithi117 ceph-mon[36002]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:48:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:53 smithi117 ceph-mon[36002]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T15:48:53.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:53 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:53 smithi120 ceph-mon[36826]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:48:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:53 smithi120 ceph-mon[36826]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T15:48:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:53 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:53 smithi023 ceph-mon[32448]: from='osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi023", "root=default"]}]': finished 2024-03-28T15:48:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:53 smithi023 ceph-mon[32448]: osdmap e67: 10 total, 9 up, 10 in 2024-03-28T15:48:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:53 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115832 s, 905 MB/s 2024-03-28T15:48:53.689 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-03-28T15:48:54.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-03-28T15:48:54.397 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:48:54.397 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:48:54.397 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:54.397 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:54.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] boot 2024-03-28T15:48:54.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T15:48:54.398 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:54.529 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] boot 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T15:48:54.569 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:48:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:48:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 804 GiB / 805 GiB avail 2024-03-28T15:48:54.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:54.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: osd.9 [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] boot 2024-03-28T15:48:54.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: osdmap e68: 10 total, 10 up, 10 in 2024-03-28T15:48:54.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:55.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:55 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:55.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:55 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: osdmap e69: 10 total, 10 up, 10 in 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:48:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:55.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:55.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:48:55.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:55 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 -- 172.21.15.117:0/2827613374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 msgr2=0x7fc148101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 --2- 172.21.15.117:0/2827613374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148101510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc14000b0d0 tx=0x7fc14000b3e0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 -- 172.21.15.117:0/2827613374 shutdown_connections 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 --2- 172.21.15.117:0/2827613374 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc148102c90 0x7fc148109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 --2- 172.21.15.117:0/2827613374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148101510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 --2- 172.21.15.117:0/2827613374 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1481022f0 0x7fc148102750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 -- 172.21.15.117:0/2827613374 >> 172.21.15.117:0/2827613374 conn(0x7fc1480fc6b0 msgr2=0x7fc1480fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 -- 172.21.15.117:0/2827613374 shutdown_connections 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.135+0000 7fc150694700 1 -- 172.21.15.117:0/2827613374 wait complete. 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 Processor -- start 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 -- start start 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1481022f0 0x7fc148197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 0x7fc14819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14810cce0 con 0x7fc1481022f0 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14810cb60 con 0x7fc1481010f0 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc150694700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc14810ce60 con 0x7fc148102c90 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39950/0 (socket says 172.21.15.117:39950) 2024-03-28T15:48:56.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 -- 172.21.15.117:0/2609230722 learned_addr learned my addr 172.21.15.117:0/2609230722 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14ec31700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 0x7fc14819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14dc2f700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1481022f0 0x7fc148197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 -- 172.21.15.117:0/2609230722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 msgr2=0x7fc14819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 0x7fc14819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 -- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1481022f0 msgr2=0x7fc148197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1481022f0 0x7fc148197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 -- 172.21.15.117:0/2609230722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc14819ef80 con 0x7fc1481010f0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14ec31700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 0x7fc14819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc14e430700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc140000f80 tx=0x7fc140007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.136+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140021020 con 0x7fc1481010f0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.137+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc14001d950 con 0x7fc1481010f0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.137+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14819f210 con 0x7fc1481010f0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.137+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc140005510 con 0x7fc1481010f0 2024-03-28T15:48:56.153 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.137+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc14819f750 con 0x7fc1481010f0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.138+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc148068e50 con 0x7fc1481010f0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.139+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc140024020 con 0x7fc1481010f0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.139+0000 7fc13f7fe700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 0x7fc1340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.139+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fc14000f020 con 0x7fc1481010f0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.139+0000 7fc14dc2f700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 0x7fc1340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.140+0000 7fc14dc2f700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 0x7fc1340a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc138005950 tx=0x7fc1380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:48:56.154 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.144+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc14009cea0 con 0x7fc1481010f0 2024-03-28T15:48:56.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:48:56.348+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1480625d0 con 0x7fc1340a2890 2024-03-28T15:48:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:56 smithi117 ceph-mon[36002]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:56 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi023 2024-03-28T15:48:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:56 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T15:48:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:56 smithi120 ceph-mon[36826]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:56 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi023 2024-03-28T15:48:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:56 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T15:48:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:56 smithi023 ceph-mon[32448]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:56 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi023 2024-03-28T15:48:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:56 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 4272M 2024-03-28T15:48:57.508 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:57.508 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:57.508 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:57.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:57.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:57.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:48:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:48:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:48:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:58 smithi117 ceph-mon[36002]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:58 smithi117 ceph-mon[36002]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:58 smithi120 ceph-mon[36826]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:58 smithi120 ceph-mon[36826]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:58 smithi023 ceph-mon[32448]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:48:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:58 smithi023 ceph-mon[32448]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:59 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/2783003560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:59 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:59 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]': finished 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:59 smithi117 ceph-mon[36002]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:48:59 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:48:59.571 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:59 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/2783003560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:59 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:59 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]': finished 2024-03-28T15:48:59.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:59 smithi023 ceph-mon[32448]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T15:48:59.572 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:48:59 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:48:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:59 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/2783003560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:59 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]: dispatch 2024-03-28T15:48:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:59 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52537bc3-64df-47b7-85c8-6b32eddd8296"}]': finished 2024-03-28T15:48:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:59 smithi120 ceph-mon[36826]: osdmap e70: 11 total, 10 up, 11 in 2024-03-28T15:48:59.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:48:59 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:00.587 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:00 smithi023 ceph-mon[32448]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:00.587 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:00 smithi023 ceph-mon[32448]: from='client.? 172.21.15.117:0/825529430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:00 smithi120 ceph-mon[36826]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:00 smithi120 ceph-mon[36826]: from='client.? 172.21.15.117:0/825529430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:00 smithi117 ceph-mon[36002]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:00 smithi117 ceph-mon[36002]: from='client.? 172.21.15.117:0/825529430' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:01.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:01 smithi023 ceph-mon[32448]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:01.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:01 smithi117 ceph-mon[36002]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:01.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:01 smithi120 ceph-mon[36826]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:03 smithi117 ceph-mon[36002]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:03.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:03 smithi023 ceph-mon[32448]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:03 smithi120 ceph-mon[36826]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:05 smithi117 ceph-mon[36002]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:05 smithi023 ceph-mon[32448]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:05 smithi120 ceph-mon[36826]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:07 smithi023 ceph-mon[32448]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:49:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:07 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:07 smithi023 ceph-mon[32448]: Deploying daemon osd.10 on smithi117 2024-03-28T15:49:07.681 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:07 smithi117 ceph-mon[36002]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:07.681 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:49:07.681 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:07 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:07.681 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:07 smithi117 ceph-mon[36002]: Deploying daemon osd.10 on smithi117 2024-03-28T15:49:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:07 smithi120 ceph-mon[36826]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-28T15:49:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:07 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:07 smithi120 ceph-mon[36826]: Deploying daemon osd.10 on smithi117 2024-03-28T15:49:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:09 smithi117 ceph-mon[36002]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:09.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:09 smithi023 ceph-mon[32448]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:09 smithi120 ceph-mon[36826]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:11.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:11 smithi023 ceph-mon[32448]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:11.707 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:11 smithi117 ceph-mon[36002]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:11.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:11 smithi120 ceph-mon[36826]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:13.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:13.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:13.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:13.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:14.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:13 smithi117 ceph-mon[36002]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:14.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:13 smithi120 ceph-mon[36826]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:13 smithi023 ceph-mon[32448]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:15.719 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 10 on host 'smithi117' 2024-03-28T15:49:15.720 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.719+0000 7fc13f7fe700 1 -- 172.21.15.117:0/2609230722 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc1480625d0 con 0x7fc1340a2890 2024-03-28T15:49:15.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 msgr2=0x7fc1340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:15.721 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 0x7fc1340a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc138005950 tx=0x7fc1380058e0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 msgr2=0x7fc148197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc140000f80 tx=0x7fc140007dc0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 shutdown_connections 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1340a2890 0x7fc1340a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc148102c90 0x7fc14819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1481010f0 0x7fc148197320 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 --2- 172.21.15.117:0/2609230722 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1481022f0 0x7fc148197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 >> 172.21.15.117:0/2609230722 conn(0x7fc1480fc6b0 msgr2=0x7fc148105670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 shutdown_connections 2024-03-28T15:49:15.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-28T15:49:15.721+0000 7fc150694700 1 -- 172.21.15.117:0/2609230722 wait complete. 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:16.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:15 smithi117 ceph-mon[36002]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:15 smithi120 ceph-mon[36826]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.379 DEBUG:teuthology.orchestra.run.smithi117:osd.10> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.10.service 2024-03-28T15:49:16.381 INFO:tasks.cephadm:Deploying osd.11 on smithi120 with /dev/vg_nvme/lv_1... 2024-03-28T15:49:16.381 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:16.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:15 smithi023 ceph-mon[32448]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-28T15:49:16.623 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:17 smithi117 ceph-mon[36002]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:18.039 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:18.039 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:49:18.040 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:18.040 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T15:49:18.040 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:18.040 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:17 smithi120 ceph-mon[36826]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:18.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:18.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-28T15:49:18.058 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: from='osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:18.059 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: osdmap e71: 11 total, 10 up, 11 in 2024-03-28T15:49:18.059 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:18.059 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:17 smithi023 ceph-mon[32448]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-03-28T15:49:18.571 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T15:49:18.156+0000 7fee76ff8700 -1 osd.10 0 waiting for initial osdmap 2024-03-28T15:49:18.571 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T15:49:18.170+0000 7fee6ede3700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr: stderr: 10+0 records in 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:10+0 records out 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0119178 s, 880 MB/s 2024-03-28T15:49:18.757 INFO:teuthology.orchestra.run.smithi120.stderr:--> Zapping successful for: 2024-03-28T15:49:18.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-mon[36002]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:49:18.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-mon[36002]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T15:49:18.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:18.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:18.988 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:18 smithi117 ceph-mon[36002]: from='osd.10 ' entity='osd.10' 2024-03-28T15:49:19.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:18 smithi120 ceph-mon[36826]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:49:19.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:18 smithi120 ceph-mon[36826]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T15:49:19.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:18 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:18 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:18 smithi120 ceph-mon[36826]: from='osd.10 ' entity='osd.10' 2024-03-28T15:49:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:18 smithi023 ceph-mon[32448]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-03-28T15:49:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:18 smithi023 ceph-mon[32448]: osdmap e72: 11 total, 10 up, 11 in 2024-03-28T15:49:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:18 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:18 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:18 smithi023 ceph-mon[32448]: from='osd.10 ' entity='osd.10' 2024-03-28T15:49:19.553 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi120:vg_nvme/lv_1 2024-03-28T15:49:19.788 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:49:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:49:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:49:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] boot 2024-03-28T15:49:19.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T15:49:19.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:19 smithi120 ceph-mon[36826]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] boot 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T15:49:19.934 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:19.935 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:19 smithi117 ceph-mon[36002]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 894 GiB / 894 GiB avail 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: osd.10 [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] boot 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: osdmap e73: 11 total, 11 up, 11 in 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:49:20.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:19 smithi023 ceph-mon[32448]: osdmap e74: 11 total, 11 up, 11 in 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi117 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-28T15:49:21.060 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.061 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:21.061 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:21.061 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.061 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:20 smithi120 ceph-mon[36826]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi117 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:20 smithi117 ceph-mon[36002]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi117 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 4270M 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:21.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:20 smithi023 ceph-mon[32448]: osdmap e75: 11 total, 11 up, 11 in 2024-03-28T15:49:21.442 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 -- 172.21.15.120:0/3894224431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102cb0 msgr2=0x7f4498109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 --2- 172.21.15.120:0/3894224431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102cb0 0x7f4498109560 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f449000b0a0 tx=0x7f449000b3b0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 -- 172.21.15.120:0/3894224431 shutdown_connections 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 --2- 172.21.15.120:0/3894224431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102cb0 0x7f4498109560 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 --2- 172.21.15.120:0/3894224431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498101110 0x7f4498101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 --2- 172.21.15.120:0/3894224431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498102310 0x7f4498102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.441+0000 7f449d3cd700 1 -- 172.21.15.120:0/3894224431 >> 172.21.15.120:0/3894224431 conn(0x7f44980fc6b0 msgr2=0x7f44980feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- 172.21.15.120:0/3894224431 shutdown_connections 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- 172.21.15.120:0/3894224431 wait complete. 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 Processor -- start 2024-03-28T15:49:21.443 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- start start 2024-03-28T15:49:21.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498101110 0x7f449819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:21.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:21.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498102cb0 0x7f44981a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:21.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810cdf0 con 0x7f4498102cb0 2024-03-28T15:49:21.446 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810cc70 con 0x7f4498101110 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.442+0000 7f449d3cd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449810cf70 con 0x7f4498102310 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:46180/0 (socket says 172.21.15.120:46180) 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 -- 172.21.15.120:0/2657459343 learned_addr learned my addr 172.21.15.120:0/2657459343 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44977fe700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498102cb0 0x7f44981a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f4496ffd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498101110 0x7f449819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 -- 172.21.15.120:0/2657459343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498101110 msgr2=0x7f449819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498101110 0x7f449819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 -- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498102cb0 msgr2=0x7f44981a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498102cb0 0x7f44981a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:21.447 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 -- 172.21.15.120:0/2657459343 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44981a23b0 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f44967fc700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f448c00b560 tx=0x7f448c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c015020 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44981a26a0 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f448c011d10 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c010e50 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.443+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44981a2cb0 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.444+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4498068e50 con 0x7f4498102310 2024-03-28T15:49:21.448 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.445+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f448c018020 con 0x7f4498102310 2024-03-28T15:49:21.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.446+0000 7f447ffff700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 0x7f44800a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:21.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.446+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f448c0cdbe0 con 0x7f4498102310 2024-03-28T15:49:21.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.446+0000 7f4496ffd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 0x7f44800a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:21.449 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.447+0000 7f4496ffd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 0x7f44800a4ef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:21.451 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.450+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f448c01f020 con 0x7f4498102310 2024-03-28T15:49:21.657 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:21.656+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44980fe840 con 0x7f44800a2a30 2024-03-28T15:49:22.476 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:22 smithi120 ceph-mon[36826]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:22.477 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:22 smithi120 ceph-mon[36826]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T15:49:22.477 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:49:22.477 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:49:22.477 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:22 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:22 smithi117 ceph-mon[36002]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:22 smithi117 ceph-mon[36002]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T15:49:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:49:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:49:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:22 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:22 smithi023 ceph-mon[32448]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:22 smithi023 ceph-mon[32448]: osdmap e76: 11 total, 11 up, 11 in 2024-03-28T15:49:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-28T15:49:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-28T15:49:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:22 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:23 smithi120 ceph-mon[36826]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:23 smithi120 ceph-mon[36826]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:49:23.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:23 smithi117 ceph-mon[36002]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:23.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:23 smithi117 ceph-mon[36002]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:49:23.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:23 smithi023 ceph-mon[32448]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi120:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:23.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:23 smithi023 ceph-mon[32448]: osdmap e77: 11 total, 11 up, 11 in 2024-03-28T15:49:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:24 smithi117 ceph-mon[36002]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-03-28T15:49:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:24 smithi120 ceph-mon[36826]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-03-28T15:49:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:24 smithi023 ceph-mon[32448]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 159 KiB/s, 0 objects/s recovering 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/2479836404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]': finished 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:25.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:25 smithi117 ceph-mon[36002]: from='client.? 172.21.15.120:0/1177762935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/2479836404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]': finished 2024-03-28T15:49:25.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T15:49:25.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:25.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:25 smithi120 ceph-mon[36826]: from='client.? 172.21.15.120:0/1177762935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/2479836404' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]: dispatch 2024-03-28T15:49:25.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7c32944-dd01-42b8-9b33-5aed77590035"}]': finished 2024-03-28T15:49:25.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: osdmap e78: 12 total, 11 up, 12 in 2024-03-28T15:49:25.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:25.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:25 smithi023 ceph-mon[32448]: from='client.? 172.21.15.120:0/1177762935' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-28T15:49:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:26 smithi117 ceph-mon[36002]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-28T15:49:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:26 smithi120 ceph-mon[36826]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-28T15:49:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:26 smithi023 ceph-mon[32448]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-28T15:49:28.476 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:28 smithi120 ceph-mon[36826]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:49:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:28 smithi117 ceph-mon[36002]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:49:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:28 smithi023 ceph-mon[32448]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-28T15:49:30.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:30 smithi117 ceph-mon[36002]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T15:49:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:30 smithi120 ceph-mon[36826]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T15:49:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:30 smithi023 ceph-mon[32448]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-28T15:49:32.539 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:32 smithi120 ceph-mon[36826]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:49:32.539 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:32 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:49:32.539 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:32 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:32 smithi117 ceph-mon[36002]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:49:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:32 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:49:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:32 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:32 smithi023 ceph-mon[32448]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-28T15:49:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:32 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-28T15:49:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:32 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:33.527 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:33 smithi120 ceph-mon[36826]: Deploying daemon osd.11 on smithi120 2024-03-28T15:49:33.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:33 smithi117 ceph-mon[36002]: Deploying daemon osd.11 on smithi120 2024-03-28T15:49:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:33 smithi023 ceph-mon[32448]: Deploying daemon osd.11 on smithi120 2024-03-28T15:49:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:34 smithi117 ceph-mon[36002]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:34 smithi120 ceph-mon[36826]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:34 smithi023 ceph-mon[32448]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:36.457 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:36 smithi120 ceph-mon[36826]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:36 smithi117 ceph-mon[36002]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:36 smithi023 ceph-mon[32448]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:38 smithi120 ceph-mon[36826]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:38 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.435 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:38 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.436 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:38 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:38 smithi117 ceph-mon[36002]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:38 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:38 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:38 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:38 smithi023 ceph-mon[32448]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:38 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:38 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:38 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:40.491 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:40 smithi120 ceph-mon[36826]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:40 smithi117 ceph-mon[36002]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:40 smithi023 ceph-mon[32448]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:41.827 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.826+0000 7f447ffff700 1 -- 172.21.15.120:0/2657459343 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f44980fe840 con 0x7f44800a2a30 2024-03-28T15:49:41.827 INFO:teuthology.orchestra.run.smithi120.stdout:Created osd(s) 11 on host 'smithi120' 2024-03-28T15:49:41.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 msgr2=0x7f44800a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:41.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 0x7f44800a4ef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4488006fd0 tx=0x7f4488008040 comp rx=0 tx=0).stop 2024-03-28T15:49:41.829 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 msgr2=0x7f449819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f448c00b560 tx=0x7f448c00b870 comp rx=0 tx=0).stop 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 shutdown_connections 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f44800a2a30 0x7f44800a4ef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4498102cb0 0x7f44981a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4498101110 0x7f449819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 --2- 172.21.15.120:0/2657459343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4498102310 0x7f449819ac90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 >> 172.21.15.120:0/2657459343 conn(0x7f44980fc6b0 msgr2=0x7f44980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 shutdown_connections 2024-03-28T15:49:41.830 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:41.829+0000 7f449d3cd700 1 -- 172.21.15.120:0/2657459343 wait complete. 2024-03-28T15:49:42.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:41 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:42.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:41 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-28T15:49:42.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:42.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:41 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:42.506 DEBUG:teuthology.orchestra.run.smithi120:osd.11> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.11.service 2024-03-28T15:49:42.509 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-28T15:49:42.509 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-28T15:49:42.764 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:49:43.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:43 smithi023 ceph-mon[32448]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T15:49:43.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:43 smithi023 ceph-mon[32448]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T15:49:43.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:43 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:43.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:43 smithi023 ceph-mon[32448]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:43 smithi023 ceph-mon[32448]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:43 smithi117 ceph-mon[36002]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T15:49:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:43 smithi117 ceph-mon[36002]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T15:49:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:43 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:43 smithi117 ceph-mon[36002]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:43 smithi117 ceph-mon[36002]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-mon[36826]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-28T15:49:43.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-mon[36826]: osdmap e79: 12 total, 11 up, 12 in 2024-03-28T15:49:43.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:43.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-mon[36826]: from='osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-mon[36826]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]: dispatch 2024-03-28T15:49:43.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T15:49:43.292+0000 7f55fb179700 -1 osd.11 0 waiting for initial osdmap 2024-03-28T15:49:43.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 15:49:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T15:49:43.305+0000 7f55f6f6c700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- 172.21.15.23:0/1639560464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 msgr2=0x7fed48102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/1639560464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed48102790 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fed38009a40 tx=0x7fed38009d50 comp rx=0 tx=0).stop 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- 172.21.15.23:0/1639560464 shutdown_connections 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/1639560464 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48102cd0 0x7fed48109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/1639560464 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48101130 0x7fed48101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/1639560464 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed48102790 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- 172.21.15.23:0/1639560464 >> 172.21.15.23:0/1639560464 conn(0x7fed480fc6f0 msgr2=0x7fed480feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:44.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- 172.21.15.23:0/1639560464 shutdown_connections 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- 172.21.15.23:0/1639560464 wait complete. 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 Processor -- start 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.403+0000 7fed4fa4b700 1 -- start start 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48101130 0x7fed48196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 0x7fed4819d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4810bbb0 con 0x7fed48102330 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4810ba30 con 0x7fed48101130 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4810bd30 con 0x7fed48102cd0 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:44.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4dfe8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 0x7fed4819d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55726/0 (socket says 172.21.15.23:55726) 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4dfe8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 0x7fed4819d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:32836/0 (socket says 172.21.15.23:32836) 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4d7e7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48101130 0x7fed48196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 -- 172.21.15.23:0/2320321017 learned_addr learned my addr 172.21.15.23:0/2320321017 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 -- 172.21.15.23:0/2320321017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 msgr2=0x7fed4819d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 0x7fed4819d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 -- 172.21.15.23:0/2320321017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48101130 msgr2=0x7fed48196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48101130 0x7fed48196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 -- 172.21.15.23:0/2320321017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4819ddc0 con 0x7fed48102330 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4cfe6700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed380094e0 tx=0x7fed38005660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed38026020 con 0x7fed48102330 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4819e050 con 0x7fed48102330 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed38005da0 con 0x7fed48102330 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.404+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed380226e0 con 0x7fed48102330 2024-03-28T15:49:44.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.405+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4819e500 con 0x7fed48102330 2024-03-28T15:49:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.405+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed48068e50 con 0x7fed48102330 2024-03-28T15:49:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.407+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fed38029020 con 0x7fed48102330 2024-03-28T15:49:44.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.407+0000 7fed3e7fc700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 0x7fed340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.407+0000 7fed4d7e7700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 0x7fed340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.407+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed380d4040 con 0x7fed48102330 2024-03-28T15:49:44.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.407+0000 7fed4d7e7700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 0x7fed340a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fed48102190 tx=0x7fed44007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.411+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fed38030020 con 0x7fed48102330 2024-03-28T15:49:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:44 smithi117 ceph-mon[36002]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:44 smithi117 ceph-mon[36002]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:49:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:44 smithi117 ceph-mon[36002]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T15:49:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:44 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:44 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:44 smithi120 ceph-mon[36826]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:44 smithi120 ceph-mon[36826]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:49:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:44 smithi120 ceph-mon[36826]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T15:49:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:44 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:44 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.615+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fed4802da30 con 0x7fed48102330 2024-03-28T15:49:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:44 smithi023 ceph-mon[32448]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:44 smithi023 ceph-mon[32448]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi120", "root=default"]}]': finished 2024-03-28T15:49:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:44 smithi023 ceph-mon[32448]: osdmap e80: 12 total, 11 up, 12 in 2024-03-28T15:49:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:44 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:44 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:44.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.616+0000 7fed3e7fc700 1 -- 172.21.15.23:0/2320321017 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fed380a1610 con 0x7fed48102330 2024-03-28T15:49:44.617 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:49:44.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 msgr2=0x7fed340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:44.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 0x7fed340a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fed48102190 tx=0x7fed44007400 comp rx=0 tx=0).stop 2024-03-28T15:49:44.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 msgr2=0x7fed481966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fed380094e0 tx=0x7fed38005660 comp rx=0 tx=0).stop 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 shutdown_connections 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fed340a2890 0x7fed340a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fed48102cd0 0x7fed4819d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed48101130 0x7fed48196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 --2- 172.21.15.23:0/2320321017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fed48102330 0x7fed481966a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 >> 172.21.15.23:0/2320321017 conn(0x7fed480fc6f0 msgr2=0x7fed48105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 shutdown_connections 2024-03-28T15:49:44.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:44.618+0000 7fed4fa4b700 1 -- 172.21.15.23:0/2320321017 wait complete. 2024-03-28T15:49:45.383 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711640984,"num_in_osds":12,"osd_in_since":1711640964,"num_remapped_pgs":0} 2024-03-28T15:49:45.383 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: purged_snaps scrub starts 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: purged_snaps scrub ok 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] boot 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:45.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:45 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2320321017' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T15:49:45.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: purged_snaps scrub starts 2024-03-28T15:49:45.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: purged_snaps scrub ok 2024-03-28T15:49:45.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] boot 2024-03-28T15:49:45.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T15:49:45.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:45.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:45 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2320321017' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T15:49:45.624 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:49:45.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: purged_snaps scrub starts 2024-03-28T15:49:45.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: purged_snaps scrub ok 2024-03-28T15:49:45.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: osd.11 [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] boot 2024-03-28T15:49:45.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: osdmap e81: 12 total, 12 up, 12 in 2024-03-28T15:49:45.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:49:45.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2320321017' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-28T15:49:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:46.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:46.425 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:46 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:46.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:46.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:46 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: osdmap e82: 12 total, 12 up, 12 in 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:46.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:46.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:46 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:47.360 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:47 smithi120 ceph-mon[36826]: Detected new or changed devices on smithi120 2024-03-28T15:49:47.361 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:47 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-28T15:49:47.385 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:47 smithi023 ceph-mon[32448]: Detected new or changed devices on smithi120 2024-03-28T15:49:47.385 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:47 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- 172.21.15.23:0/2307279096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a8074900 msgr2=0x7f17a8074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 --2- 172.21.15.23:0/2307279096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a8074900 0x7f17a8074d80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f179c009990 tx=0x7f179c009ca0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- 172.21.15.23:0/2307279096 shutdown_connections 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 --2- 172.21.15.23:0/2307279096 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 0x7f17a8115240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 --2- 172.21.15.23:0/2307279096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a80761a0 0x7f17a80743c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 --2- 172.21.15.23:0/2307279096 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a8074900 0x7f17a8074d80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- 172.21.15.23:0/2307279096 >> 172.21.15.23:0/2307279096 conn(0x7f17a8079220 msgr2=0x7f17a8079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- 172.21.15.23:0/2307279096 shutdown_connections 2024-03-28T15:49:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- 172.21.15.23:0/2307279096 wait complete. 2024-03-28T15:49:47.390 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 Processor -- start 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.388+0000 7f17b0109700 1 -- start start 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a8074900 0x7f17a819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 0x7f17a81a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8118af0 con 0x7f17a80761a0 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8118970 con 0x7f17a8074900 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8118c70 con 0x7f17a8112e50 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55750/0 (socket says 172.21.15.23:55750) 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 -- 172.21.15.23:0/843671528 learned_addr learned my addr 172.21.15.23:0/843671528 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ae6a6700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 0x7f17a81a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17adea5700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a8074900 0x7f17a819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:47.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 -- 172.21.15.23:0/843671528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 msgr2=0x7f17a81a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 0x7f17a81a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 -- 172.21.15.23:0/843671528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a8074900 msgr2=0x7f17a819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a8074900 0x7f17a819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 -- 172.21.15.23:0/843671528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a81a3f60 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17ad6a4700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f179c005910 tx=0x7f179c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c027070 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f179c003e40 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.389+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a81a41f0 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.390+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179c022d00 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.390+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a81abad0 con 0x7f17a80761a0 2024-03-28T15:49:47.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.390+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a8075a50 con 0x7f17a80761a0 2024-03-28T15:49:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.392+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f179c02a070 con 0x7f17a80761a0 2024-03-28T15:49:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.392+0000 7f179affd700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 0x7f17940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.392+0000 7f17adea5700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 0x7f17940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.392+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f179c0cfa40 con 0x7f17a80761a0 2024-03-28T15:49:47.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.393+0000 7f17adea5700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 0x7f17940a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f17a4005950 tx=0x7f17a400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.396+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f179c031070 con 0x7f17a80761a0 2024-03-28T15:49:47.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:47 smithi117 ceph-mon[36002]: Detected new or changed devices on smithi120 2024-03-28T15:49:47.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:47 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 4270M 2024-03-28T15:49:47.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.602+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a8068e50 con 0x7f17a80761a0 2024-03-28T15:49:47.603 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.603+0000 7f179affd700 1 -- 172.21.15.23:0/843671528 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7f17a8068e50 con 0x7f17a80761a0 2024-03-28T15:49:47.603 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:49:47.603 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:49:45.294119+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:49:47.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 msgr2=0x7f17940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:47.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 0x7f17940a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f17a4005950 tx=0x7f17a400a300 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 msgr2=0x7f17a819f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f179c005910 tx=0x7f179c0047b0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 shutdown_connections 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f17940a2890 0x7f17940a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f17a8112e50 0x7f17a81a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17a8074900 0x7f17a819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 --2- 172.21.15.23:0/843671528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17a80761a0 0x7f17a819f4e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 >> 172.21.15.23:0/843671528 conn(0x7f17a8079220 msgr2=0x7f17a8111b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 shutdown_connections 2024-03-28T15:49:47.606 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:47.605+0000 7f17b0109700 1 -- 172.21.15.23:0/843671528 wait complete. 2024-03-28T15:49:48.336 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:49:48.336 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-28T15:49:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:48 smithi117 ceph-mon[36002]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/843671528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:48.575 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:49:48.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:48 smithi120 ceph-mon[36826]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:48.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/843671528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:48 smithi023 ceph-mon[32448]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/843671528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:50.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 -- 172.21.15.23:0/1384670188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2841024a0 msgr2=0x7fe284105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/1384670188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2841024a0 0x7fe284105d30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe26c0099c0 tx=0x7fe26c009cd0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 -- 172.21.15.23:0/1384670188 shutdown_connections 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/1384670188 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe284106410 0x7fe28410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/1384670188 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 0x7fe284101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/1384670188 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe2841024a0 0x7fe284105d30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 -- 172.21.15.23:0/1384670188 >> 172.21.15.23:0/1384670188 conn(0x7fe284077690 msgr2=0x7fe2840feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 -- 172.21.15.23:0/1384670188 shutdown_connections 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.199+0000 7fe28a1c8700 1 -- 172.21.15.23:0/1384670188 wait complete. 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 Processor -- start 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 -- start start 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 0x7fe284199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:50.212 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2841024a0 0x7fe284199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2841113f0 con 0x7fe284106410 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284111270 con 0x7fe284101af0 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe28a1c8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe284111570 con 0x7fe2841024a0 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe283fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe283fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46238/0 (socket says 172.21.15.23:46238) 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe283fff700 1 -- 172.21.15.23:0/56664774 learned_addr learned my addr 172.21.15.23:0/56664774 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.200+0000 7fe282ffd700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2841024a0 0x7fe284199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe2837fe700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 0x7fe284199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 -- 172.21.15.23:0/56664774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2841024a0 msgr2=0x7fe284199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2841024a0 0x7fe284199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 -- 172.21.15.23:0/56664774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 msgr2=0x7fe284199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 0x7fe284199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.213 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 -- 172.21.15.23:0/56664774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2841a4d50 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe283fff700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c970 tx=0x7fe27c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c004a30 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2841a4f80 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe27c005500 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27c00ecf0 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.201+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2841a53e0 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.202+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe284068e50 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.203+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe27c005000 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.204+0000 7fe280ff9700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 0x7fe2700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.204+0000 7fe2837fe700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 0x7fe2700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.205+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe27c0ce280 con 0x7fe284106410 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.205+0000 7fe2837fe700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 0x7fe2700a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe27400f440 tx=0x7fe2740079e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:50.214 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.208+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe27c014030 con 0x7fe284106410 2024-03-28T15:49:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.412+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe28402dbf0 con 0x7fe284106410 2024-03-28T15:49:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.413+0000 7fe280ff9700 1 -- 172.21.15.23:0/56664774 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe27c09b7c0 con 0x7fe284106410 2024-03-28T15:49:50.413 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 msgr2=0x7fe2700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 0x7fe2700a4d50 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe27400f440 tx=0x7fe2740079e0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 msgr2=0x7fe2841a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe27c00c970 tx=0x7fe27c00cc80 comp rx=0 tx=0).stop 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 shutdown_connections 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2700a2890 0x7fe2700a4d50 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe284106410 0x7fe2841a4810 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe284101af0 0x7fe284199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 --2- 172.21.15.23:0/56664774 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe2841024a0 0x7fe284199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:50.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 >> 172.21.15.23:0/56664774 conn(0x7fe284077690 msgr2=0x7fe284104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:50.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 shutdown_connections 2024-03-28T15:49:50.416 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:50.415+0000 7fe28a1c8700 1 -- 172.21.15.23:0/56664774 wait complete. 2024-03-28T15:49:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:50 smithi117 ceph-mon[36002]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:50.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:50 smithi120 ceph-mon[36826]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:50 smithi023 ceph-mon[32448]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.a on smithi023 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.d on smithi023 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.g on smithi023 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.j on smithi023 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.b on smithi117 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.e on smithi117 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.h on smithi117 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.k on smithi117 2024-03-28T15:49:51.001 INFO:tasks.cephadm:Adding mds.c on smithi120 2024-03-28T15:49:51.002 INFO:tasks.cephadm:Adding mds.f on smithi120 2024-03-28T15:49:51.002 INFO:tasks.cephadm:Adding mds.i on smithi120 2024-03-28T15:49:51.002 INFO:tasks.cephadm:Adding mds.l on smithi120 2024-03-28T15:49:51.002 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l' 2024-03-28T15:49:51.250 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T15:49:51.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/56664774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:51.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/56664774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:51.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/56664774' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:52 smithi120 ceph-mon[36826]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:52 smithi023 ceph-mon[32448]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:52 smithi117 ceph-mon[36002]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 -- 172.21.15.120:0/4064991939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 msgr2=0x7f982c10a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 --2- 172.21.15.120:0/4064991939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c10a7d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f982400b0a0 tx=0x7f982400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 -- 172.21.15.120:0/4064991939 shutdown_connections 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 --2- 172.21.15.120:0/4064991939 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c10a7d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 --2- 172.21.15.120:0/4064991939 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c1037e0 0x7f982c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 --2- 172.21.15.120:0/4064991939 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c100260 0x7f982c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 -- 172.21.15.120:0/4064991939 >> 172.21.15.120:0/4064991939 conn(0x7f982c0fbe20 msgr2=0x7f982c0fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.892+0000 7f98335ca700 1 -- 172.21.15.120:0/4064991939 shutdown_connections 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 -- 172.21.15.120:0/4064991939 wait complete. 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 Processor -- start 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 -- start start 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c100260 0x7f982c198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:52.895 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1037e0 0x7f982c19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c10dde0 con 0x7f982c1037e0 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c10dc60 con 0x7f982c100260 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f98335ca700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c10df60 con 0x7f982c100c20 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f9830b65700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f9830b65700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.120:53568/0 (socket says 172.21.15.120:53568) 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f9830b65700 1 -- 172.21.15.120:0/2537529393 learned_addr learned my addr 172.21.15.120:0/2537529393 (peer_addr_for_me v2:172.21.15.120:0/0) 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f9831b67700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1037e0 0x7f982c19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.893+0000 7f9831366700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c100260 0x7f982c198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 -- 172.21.15.120:0/2537529393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c100260 msgr2=0x7f982c198330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c100260 0x7f982c198330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 -- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1037e0 msgr2=0x7f982c19f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:52.896 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1037e0 0x7f982c19f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 -- 172.21.15.120:0/2537529393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c19ff90 con 0x7f982c100c20 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f9830b65700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f982800c9a0 tx=0x7f982800cd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9828004220 con 0x7f982c100c20 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9828007d40 con 0x7f982c100c20 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c1a0280 con 0x7f982c100c20 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982800f450 con 0x7f982c100c20 2024-03-28T15:49:52.897 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.894+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c1a0810 con 0x7f982c100c20 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.895+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c068e50 con 0x7f982c100c20 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.896+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9828007860 con 0x7f982c100c20 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.896+0000 7f98227fc700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 0x7f98180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.897+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f98280ce4f0 con 0x7f982c100c20 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.897+0000 7f9831366700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 0x7f98180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:52.898 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.897+0000 7f9831366700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 0x7f98180a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f981c005fd0 tx=0x7f981c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:52.900 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:52.900+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9828014030 con 0x7f982c100c20 2024-03-28T15:49:53.103 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.102+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f982c0627d0 con 0x7f98180a2910 2024-03-28T15:49:53.111 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.111+0000 7f98227fc700 1 -- 172.21.15.120:0/2537529393 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f982c0627d0 con 0x7f98180a2910 2024-03-28T15:49:53.111 INFO:teuthology.orchestra.run.smithi120.stdout:Scheduled mds.all update... 2024-03-28T15:49:53.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 msgr2=0x7f98180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:53.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 0x7f98180a4dd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f981c005fd0 tx=0x7f981c00a560 comp rx=0 tx=0).stop 2024-03-28T15:49:53.113 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 msgr2=0x7f982c198870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f982800c9a0 tx=0x7f982800cd60 comp rx=0 tx=0).stop 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 shutdown_connections 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98180a2910 0x7f98180a4dd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f982c1037e0 0x7f982c19f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f982c100260 0x7f982c198330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 --2- 172.21.15.120:0/2537529393 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f982c100c20 0x7f982c198870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 >> 172.21.15.120:0/2537529393 conn(0x7f982c0fbe20 msgr2=0x7f982c1067b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 shutdown_connections 2024-03-28T15:49:53.114 INFO:teuthology.orchestra.run.smithi120.stderr:2024-03-28T15:49:53.113+0000 7f98335ca700 1 -- 172.21.15.120:0/2537529393 wait complete. 2024-03-28T15:49:53.826 DEBUG:teuthology.orchestra.run.smithi023:mds.a> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.a.service 2024-03-28T15:49:53.828 DEBUG:teuthology.orchestra.run.smithi023:mds.d> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.d.service 2024-03-28T15:49:53.871 DEBUG:teuthology.orchestra.run.smithi023:mds.g> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.g.service 2024-03-28T15:49:53.914 DEBUG:teuthology.orchestra.run.smithi023:mds.j> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.j.service 2024-03-28T15:49:53.957 DEBUG:teuthology.orchestra.run.smithi117:mds.b> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.b.service 2024-03-28T15:49:53.959 DEBUG:teuthology.orchestra.run.smithi117:mds.e> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.e.service 2024-03-28T15:49:54.002 DEBUG:teuthology.orchestra.run.smithi117:mds.h> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.h.service 2024-03-28T15:49:54.045 DEBUG:teuthology.orchestra.run.smithi117:mds.k> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.k.service 2024-03-28T15:49:54.090 DEBUG:teuthology.orchestra.run.smithi120:mds.c> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.c.service 2024-03-28T15:49:54.092 DEBUG:teuthology.orchestra.run.smithi120:mds.f> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.f.service 2024-03-28T15:49:54.135 DEBUG:teuthology.orchestra.run.smithi120:mds.i> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.i.service 2024-03-28T15:49:54.178 DEBUG:teuthology.orchestra.run.smithi120:mds.l> sudo journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.l.service 2024-03-28T15:49:54.221 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-28T15:49:54.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:54.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:54.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:54.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:54 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:54.364 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.365 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:54.365 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.365 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:54.365 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:54.365 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:54 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: Saving service mds.all spec with placement smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:49:54.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:54.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:54.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:54.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:54 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 -- 172.21.15.23:0/3250345671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 msgr2=0x7f0060108190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 --2- 172.21.15.23:0/3250345671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f0060108190 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f005800adb0 tx=0x7f005802cff0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 -- 172.21.15.23:0/3250345671 shutdown_connections 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 --2- 172.21.15.23:0/3250345671 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f0060108190 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 --2- 172.21.15.23:0/3250345671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 0x7f00601013c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 --2- 172.21.15.23:0/3250345671 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 0x7f0060100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 -- 172.21.15.23:0/3250345671 >> 172.21.15.23:0/3250345671 conn(0x7f00600fb2c0 msgr2=0x7f00600fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 -- 172.21.15.23:0/3250345671 shutdown_connections 2024-03-28T15:49:54.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.479+0000 7f0067976640 1 -- 172.21.15.23:0/3250345671 wait complete. 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 Processor -- start 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 -- start start 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 0x7f00601957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 0x7f0060195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006010b160 con 0x7f0060101900 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006010afe0 con 0x7f0060100f40 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0067976640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006010b2e0 con 0x7f00600ffd50 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0065f13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0065f13640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46262/0 (socket says 172.21.15.23:46262) 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0065f13640 1 -- 172.21.15.23:0/1534098971 learned_addr learned my addr 172.21.15.23:0/1534098971 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0065712640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 0x7f00601957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.480+0000 7f0064f11640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 0x7f0060195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 -- 172.21.15.23:0/1534098971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 msgr2=0x7f00601957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 0x7f00601957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 -- 172.21.15.23:0/1534098971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 msgr2=0x7f0060195cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 0x7f0060195cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 -- 172.21.15.23:0/1534098971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006019d3d0 con 0x7f0060101900 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0065f13640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0058038040 tx=0x7f0058003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058042020 con 0x7f0060101900 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0058003e40 con 0x7f0060101900 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005803a040 con 0x7f0060101900 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006019d660 con 0x7f0060101900 2024-03-28T15:49:54.485 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.481+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006019dba0 con 0x7f0060101900 2024-03-28T15:49:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.482+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0060107070 con 0x7f0060101900 2024-03-28T15:49:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.484+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0058045020 con 0x7f0060101900 2024-03-28T15:49:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.484+0000 7f00567fc640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 0x7f004c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.485+0000 7f0065712640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 0x7f004c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.486 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.485+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f00580f1890 con 0x7f0060101900 2024-03-28T15:49:54.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.485+0000 7f0065712640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 0x7f004c0a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f005c005fd0 tx=0x7f005c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:54.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.488+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f005804c020 con 0x7f0060101900 2024-03-28T15:49:54.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.672+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f006002d240 con 0x7f0060101900 2024-03-28T15:49:54.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.673+0000 7f00567fc640 1 -- 172.21.15.23:0/1534098971 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7f00580bedd0 con 0x7f0060101900 2024-03-28T15:49:54.674 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:49:54.674 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:49:45.294119+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:49:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 msgr2=0x7f004c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 0x7f004c0a4d40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f005c005fd0 tx=0x7f005c00b390 comp rx=0 tx=0).stop 2024-03-28T15:49:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 msgr2=0x7f006019cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0058038040 tx=0x7f0058003c20 comp rx=0 tx=0).stop 2024-03-28T15:49:54.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 shutdown_connections 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f004c0a2880 0x7f004c0a4d40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0060101900 0x7f006019cd60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0060100f40 0x7f0060195cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 --2- 172.21.15.23:0/1534098971 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f00600ffd50 0x7f00601957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 >> 172.21.15.23:0/1534098971 conn(0x7f00600fb2c0 msgr2=0x7f0060104240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 shutdown_connections 2024-03-28T15:49:54.677 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.675+0000 7f0067976640 1 -- 172.21.15.23:0/1534098971 wait complete. 2024-03-28T15:49:54.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:49:54.687 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:49:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 -- 172.21.15.23:0/3252736698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0104a70 msgr2=0x7fd9d0106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 --2- 172.21.15.23:0/3252736698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0104a70 0x7fd9d0106e60 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd9b8009900 tx=0x7fd9b802bc30 comp rx=0 tx=0).stop 2024-03-28T15:49:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 -- 172.21.15.23:0/3252736698 shutdown_connections 2024-03-28T15:49:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 --2- 172.21.15.23:0/3252736698 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d01073a0 0x7fd9d01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 --2- 172.21.15.23:0/3252736698 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0104a70 0x7fd9d0106e60 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 --2- 172.21.15.23:0/3252736698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d0102140 0x7fd9d0104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 -- 172.21.15.23:0/3252736698 >> 172.21.15.23:0/3252736698 conn(0x7fd9d00fbb50 msgr2=0x7fd9d00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 -- 172.21.15.23:0/3252736698 shutdown_connections 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.791+0000 7fd9d5953640 1 -- 172.21.15.23:0/3252736698 wait complete. 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 Processor -- start 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 -- start start 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 0x7fd9d01000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d01073a0 0x7fd9d0100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010d410 con 0x7fd9d0102140 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010d290 con 0x7fd9d01073a0 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9d5953640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d010d590 con 0x7fd9d0104a70 2024-03-28T15:49:54.796 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46272/0 (socket says 172.21.15.23:46272) 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 -- 172.21.15.23:0/1055643759 learned_addr learned my addr 172.21.15.23:0/1055643759 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9c67fc640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 0x7fd9d01000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9cf7fe640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d01073a0 0x7fd9d0100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 -- 172.21.15.23:0/1055643759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 msgr2=0x7fd9d01000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 0x7fd9d01000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 -- 172.21.15.23:0/1055643759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d01073a0 msgr2=0x7fd9d0100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d01073a0 0x7fd9d0100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9ceffd640 1 -- 172.21.15.23:0/1055643759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d0100ec0 con 0x7fd9d0102140 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.792+0000 7fd9c67fc640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 0x7fd9d01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9ceffd640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000b4a0 tx=0x7fd9c000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0014070 con 0x7fd9d0102140 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9c0010af0 con 0x7fd9d0102140 2024-03-28T15:49:54.797 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d01a1c00 con 0x7fd9d0102140 2024-03-28T15:49:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c0017d10 con 0x7fd9d0102140 2024-03-28T15:49:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.793+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d01a2120 con 0x7fd9d0102140 2024-03-28T15:49:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.794+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d00688e0 con 0x7fd9d0102140 2024-03-28T15:49:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.796+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd9c0017070 con 0x7fd9d0102140 2024-03-28T15:49:54.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.796+0000 7fd9d4951640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 0x7fd9bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.796+0000 7fd9c67fc640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 0x7fd9bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.796+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd9c00cde80 con 0x7fd9d0102140 2024-03-28T15:49:54.799 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.797+0000 7fd9c67fc640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 0x7fd9bc0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd9b802c7b0 tx=0x7fd9b8004f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:54.800 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.800+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd9c009b3c0 con 0x7fd9d0102140 2024-03-28T15:49:54.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.982+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9d002d070 con 0x7fd9d0102140 2024-03-28T15:49:54.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.982+0000 7fd9d4951640 1 -- 172.21.15.23:0/1055643759 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9c009acc0 con 0x7fd9d0102140 2024-03-28T15:49:54.983 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:49:54.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 msgr2=0x7fd9bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 0x7fd9bc0a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd9b802c7b0 tx=0x7fd9b8004f10 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 msgr2=0x7fd9d0101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd9c000b4a0 tx=0x7fd9c000b960 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 shutdown_connections 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd9bc0a2800 0x7fd9bc0a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd9d01073a0 0x7fd9d0100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd9d0104a70 0x7fd9d01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 --2- 172.21.15.23:0/1055643759 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd9d0102140 0x7fd9d0101ab0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 >> 172.21.15.23:0/1055643759 conn(0x7fd9d00fbb50 msgr2=0x7fd9d0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 shutdown_connections 2024-03-28T15:49:54.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:54.984+0000 7fd9d5953640 1 -- 172.21.15.23:0/1055643759 wait complete. 2024-03-28T15:49:54.997 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:49:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- 172.21.15.23:0/1125463448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba01017b0 msgr2=0x7f7ba0101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 --2- 172.21.15.23:0/1125463448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba01017b0 0x7f7ba0101c30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7b88009900 tx=0x7f7b8802bc30 comp rx=0 tx=0).stop 2024-03-28T15:49:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- 172.21.15.23:0/1125463448 shutdown_connections 2024-03-28T15:49:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 --2- 172.21.15.23:0/1125463448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba0102170 0x7f7ba0108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 --2- 172.21.15.23:0/1125463448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba01017b0 0x7f7ba0101c30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 --2- 172.21.15.23:0/1125463448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01005c0 0x7f7ba01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- 172.21.15.23:0/1125463448 >> 172.21.15.23:0/1125463448 conn(0x7f7ba00fbb30 msgr2=0x7f7ba00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- 172.21.15.23:0/1125463448 shutdown_connections 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- 172.21.15.23:0/1125463448 wait complete. 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 Processor -- start 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.098+0000 7f7ba6376640 1 -- start start 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 0x7f7ba01968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 0x7f7ba0196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba010c240 con 0x7f7ba0102170 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba010c0c0 con 0x7f7ba01017b0 2024-03-28T15:49:55.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba010c3c0 con 0x7f7ba01005c0 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46286/0 (socket says 172.21.15.23:46286) 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 -- 172.21.15.23:0/2164289884 learned_addr learned my addr 172.21.15.23:0/2164289884 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9f7fe640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 0x7f7ba0196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9ffff640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 0x7f7ba01968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 -- 172.21.15.23:0/2164289884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 msgr2=0x7f7ba01968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 0x7f7ba01968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 -- 172.21.15.23:0/2164289884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 msgr2=0x7f7ba0196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 0x7f7ba0196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 -- 172.21.15.23:0/2164289884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba019e500 con 0x7f7ba0102170 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9f7fe640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 0x7f7ba0196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9ffff640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 0x7f7ba01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba4913640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800eb10 tx=0x7f7b9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:55.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98004b20 con 0x7f7ba0102170 2024-03-28T15:49:55.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba019e7f0 con 0x7f7ba0102170 2024-03-28T15:49:55.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b98004cc0 con 0x7f7ba0102170 2024-03-28T15:49:55.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba019ed60 con 0x7f7ba0102170 2024-03-28T15:49:55.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.099+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b98010760 con 0x7f7ba0102170 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.101+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba01078e0 con 0x7f7ba0102170 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.102+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7b98005020 con 0x7f7ba0102170 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.102+0000 7f7b9d7fa640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 0x7f7b8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.102+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b980cee10 con 0x7f7ba0102170 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.102+0000 7f7b9ffff640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 0x7f7b8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:55.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.103+0000 7f7b9ffff640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 0x7f7b8c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0101610 tx=0x7f7b90005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:55.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.107+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b9809c350 con 0x7f7ba0102170 2024-03-28T15:49:55.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.327+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ba00622a0 con 0x7f7ba0102170 2024-03-28T15:49:55.329 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.328+0000 7f7b9d7fa640 1 -- 172.21.15.23:0/2164289884 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f7b9809bc50 con 0x7f7ba0102170 2024-03-28T15:49:55.329 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:49:55.329 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-28T15:49:55.330 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 1 2024-03-28T15:49:55.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 msgr2=0x7f7b8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:55.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 0x7f7b8c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0101610 tx=0x7f7b90005d30 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 msgr2=0x7f7ba019de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800eb10 tx=0x7f7b9800c2d0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 shutdown_connections 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7b8c0a2800 0x7f7b8c0a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ba0102170 0x7f7ba019de90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba01017b0 0x7f7ba0196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 --2- 172.21.15.23:0/2164289884 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ba01005c0 0x7f7ba01968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 >> 172.21.15.23:0/2164289884 conn(0x7f7ba00fbb30 msgr2=0x7f7ba0104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 shutdown_connections 2024-03-28T15:49:55.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:55.330+0000 7f7ba6376640 1 -- 172.21.15.23:0/2164289884 wait complete. 2024-03-28T15:49:55.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:55 smithi023 ceph-mon[32448]: Deploying daemon mds.l on smithi120 2024-03-28T15:49:55.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:55 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1534098971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:55.341 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:55 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1055643759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:55.399 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:55 smithi120 ceph-mon[36826]: Deploying daemon mds.l on smithi120 2024-03-28T15:49:55.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:55 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1534098971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:55.400 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:55 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1055643759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:55 smithi117 ceph-mon[36002]: Deploying daemon mds.l on smithi120 2024-03-28T15:49:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:55 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1534098971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:49:55.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:55 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1055643759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:49:56.090 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:55 smithi120 systemd[1]: Starting Ceph mds.l for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:49:56.394 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:56 smithi120 ceph-mon[36826]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:56.394 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:56 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2164289884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:56.395 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 podman[66159]: 2024-03-28 15:49:56.047167899 +0000 UTC m=+0.017426391 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:49:56.395 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 podman[66159]: 2024-03-28 15:49:56.15015968 +0000 UTC m=+0.120418214 container create 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T15:49:56.395 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 podman[66159]: 2024-03-28 15:49:56.394749277 +0000 UTC m=+0.365007774 container init 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T15:49:56.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:56 smithi023 ceph-mon[32448]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:56.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:56 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2164289884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:56 smithi117 ceph-mon[36002]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:56 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2164289884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:56.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 podman[66159]: 2024-03-28 15:49:56.397358019 +0000 UTC m=+0.367616507 container start 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T15:49:56.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 ceph-mds[66179]: -- 172.21.15.120:0/3498452823 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d2e2f1a40 con 0x561d2e2c4c00 2024-03-28T15:49:56.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: starting mds.l at 2024-03-28T15:49:56.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 bash[66159]: 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c 2024-03-28T15:49:56.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 15:49:56 smithi120 systemd[1]: Started Ceph mds.l for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:49:57.342 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 -- 172.21.15.23:0/984286869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8601028d0 msgr2=0x7fe860102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 --2- 172.21.15.23:0/984286869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8601028d0 0x7fe860102d50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe854009900 tx=0x7fe85402bc30 comp rx=0 tx=0).stop 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 -- 172.21.15.23:0/984286869 shutdown_connections 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 --2- 172.21.15.23:0/984286869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe860103290 0x7fe860109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 --2- 172.21.15.23:0/984286869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8601028d0 0x7fe860102d50 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 --2- 172.21.15.23:0/984286869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 0x7fe860101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 -- 172.21.15.23:0/984286869 >> 172.21.15.23:0/984286869 conn(0x7fe8600fcc50 msgr2=0x7fe8600ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 -- 172.21.15.23:0/984286869 shutdown_connections 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.458+0000 7fe8660ac640 1 -- 172.21.15.23:0/984286869 wait complete. 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 Processor -- start 2024-03-28T15:49:57.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 -- start start 2024-03-28T15:49:57.462 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 0x7fe8601979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8601028d0 0x7fe860197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86010d360 con 0x7fe860103290 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86010d1e0 con 0x7fe8601016e0 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe8660ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe86010d4e0 con 0x7fe8601028d0 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46296/0 (socket says 172.21.15.23:46296) 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 -- 172.21.15.23:0/3911856825 learned_addr learned my addr 172.21.15.23:0/3911856825 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85f7fe640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 0x7fe8601979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85effd640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8601028d0 0x7fe860197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 -- 172.21.15.23:0/3911856825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8601028d0 msgr2=0x7fe860197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8601028d0 0x7fe860197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.459+0000 7fe85ffff640 1 -- 172.21.15.23:0/3911856825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 msgr2=0x7fe8601979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85ffff640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 0x7fe8601979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85ffff640 1 -- 172.21.15.23:0/3911856825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe86019f5f0 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85ffff640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe85800eb10 tx=0x7fe85800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858004b20 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe858004cc0 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe858010760 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe86019f8e0 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.460+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe86019fe00 con 0x7fe860103290 2024-03-28T15:49:57.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.461+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe860108a00 con 0x7fe860103290 2024-03-28T15:49:57.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.462+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe858005020 con 0x7fe860103290 2024-03-28T15:49:57.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.463+0000 7fe85cff9640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 0x7fe8480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:57.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.463+0000 7fe85f7fe640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 0x7fe8480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:57.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.463+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe8580ced50 con 0x7fe860103290 2024-03-28T15:49:57.465 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.463+0000 7fe85f7fe640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 0x7fe8480a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe860102730 tx=0x7fe850005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6832/515874083,v1:172.21.15.120:6833/515874083] up:boot 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: fsmap 1 up:standby 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.467 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.468 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:57.468 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:57.468 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:57.468 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:57 smithi117 ceph-mon[36002]: Deploying daemon mds.k on smithi117 2024-03-28T15:49:57.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.467+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8580d2030 con 0x7fe860103290 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6832/515874083,v1:172.21.15.120:6833/515874083] up:boot 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: fsmap 1 up:standby 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:57.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:57.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:57.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:57 smithi023 ceph-mon[32448]: Deploying daemon mds.k on smithi117 2024-03-28T15:49:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.689+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe86002d0c0 con 0x7fe860103290 2024-03-28T15:49:57.690 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.689+0000 7fe85cff9640 1 -- 172.21.15.23:0/3911856825 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7fe85809c290 con 0x7fe860103290 2024-03-28T15:49:57.691 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:49:57.691 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:49:57.691 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 2 2024-03-28T15:49:57.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 msgr2=0x7fe8480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:57.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 0x7fe8480a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe860102730 tx=0x7fe850005f90 comp rx=0 tx=0).stop 2024-03-28T15:49:57.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 msgr2=0x7fe86019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:57.693 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe85800eb10 tx=0x7fe85800c2d0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 shutdown_connections 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8480a2800 0x7fe8480a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe860103290 0x7fe86019ef80 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8601028d0 0x7fe860197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 --2- 172.21.15.23:0/3911856825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8601016e0 0x7fe8601979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 >> 172.21.15.23:0/3911856825 conn(0x7fe8600fcc50 msgr2=0x7fe860105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 shutdown_connections 2024-03-28T15:49:57.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:57.692+0000 7fe8660ac640 1 -- 172.21.15.23:0/3911856825 wait complete. 2024-03-28T15:49:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6832/515874083,v1:172.21.15.120:6833/515874083] up:boot 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: fsmap 1 up:standby 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:49:57.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:57 smithi120 ceph-mon[36826]: Deploying daemon mds.k on smithi117 2024-03-28T15:49:58.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:58 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3911856825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:58 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3911856825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3911856825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:49:59.022 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:58 smithi117 systemd[1]: Starting Ceph mds.k for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:49:59.319 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 podman[65361]: 2024-03-28 15:49:58.925459922 +0000 UTC m=+0.017721960 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:49:59.319 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 podman[65361]: 2024-03-28 15:49:59.066542692 +0000 UTC m=+0.158804728 container create ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:49:59 smithi117 ceph-mon[36002]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 podman[65361]: 2024-03-28 15:49:59.319281312 +0000 UTC m=+0.411543353 container init ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 podman[65361]: 2024-03-28 15:49:59.322256294 +0000 UTC m=+0.414518339 container start ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 ceph-mds[65380]: -- 172.21.15.117:0/1240624498 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563860f9fa40 con 0x563860f72800 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: starting mds.k at 2024-03-28T15:49:59.571 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 bash[65361]: ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e 2024-03-28T15:49:59.572 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 15:49:59 smithi117 systemd[1]: Started Ceph mds.k for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:49:59.705 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:49:59.728 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:49:59 smithi023 ceph-mon[32448]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:49:59.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.808+0000 7fa801977640 1 -- 172.21.15.23:0/600475462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1073a0 msgr2=0x7fa7fc1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:59.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.808+0000 7fa801977640 1 --2- 172.21.15.23:0/600475462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc1097f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4009ca0 tx=0x7fa7f402c0f0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.808+0000 7fa801977640 1 -- 172.21.15.23:0/600475462 shutdown_connections 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.808+0000 7fa801977640 1 --2- 172.21.15.23:0/600475462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc1097f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 --2- 172.21.15.23:0/600475462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc104a70 0x7fa7fc106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 --2- 172.21.15.23:0/600475462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc102140 0x7fa7fc104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 -- 172.21.15.23:0/600475462 >> 172.21.15.23:0/600475462 conn(0x7fa7fc0fbb30 msgr2=0x7fa7fc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 -- 172.21.15.23:0/600475462 shutdown_connections 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 -- 172.21.15.23:0/600475462 wait complete. 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.809+0000 7fa801977640 1 Processor -- start 2024-03-28T15:49:59.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 -- start start 2024-03-28T15:49:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc104a70 0x7fa7fc197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:59.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc10d340 con 0x7fa7fc102140 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc10d1c0 con 0x7fa7fc1073a0 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa801977640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc10d4c0 con 0x7fa7fc104a70 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54780/0 (socket says 172.21.15.23:54780) 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 -- 172.21.15.23:0/765034438 learned_addr learned my addr 172.21.15.23:0/765034438 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7fa7fc640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc104a70 0x7fa7fc197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 -- 172.21.15.23:0/765034438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc104a70 msgr2=0x7fa7fc197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc104a70 0x7fa7fc197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7fb7fe640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 -- 172.21.15.23:0/765034438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 msgr2=0x7fa7fc19eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc19eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 -- 172.21.15.23:0/765034438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc19f540 con 0x7fa7fc102140 2024-03-28T15:49:59.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7fb7fe640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa7faffd640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00c920 tx=0x7fa7ec00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.810+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec007c70 con 0x7fa7fc102140 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.811+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7ec007e10 con 0x7fa7fc102140 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.811+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7fc19f830 con 0x7fa7fc102140 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.811+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7ec018890 con 0x7fa7fc102140 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.811+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc19fd80 con 0x7fa7fc102140 2024-03-28T15:49:59.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.811+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7fc0fd620 con 0x7fa7fc102140 2024-03-28T15:49:59.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.813+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa7ec018a50 con 0x7fa7fc102140 2024-03-28T15:49:59.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.813+0000 7fa800975640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 0x7fa7e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:49:59.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.813+0000 7fa7fa7fc640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 0x7fa7e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:49:59.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.813+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa7ec0cea70 con 0x7fa7fc102140 2024-03-28T15:49:59.816 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.814+0000 7fa7fa7fc640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 0x7fa7e40a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005fd0 tx=0x7fa7f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:49:59.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:49:59.821+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa7ec014030 con 0x7fa7fc102140 2024-03-28T15:49:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:49:59 smithi120 ceph-mon[36826]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.049+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa7fc02d070 con 0x7fa7fc102140 2024-03-28T15:50:00.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.049+0000 7fa800975640 1 -- 172.21.15.23:0/765034438 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fa7ec09c040 con 0x7fa7fc102140 2024-03-28T15:50:00.051 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:00.051 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:50:00.051 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 msgr2=0x7fa7e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 0x7fa7e40a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa7f0005fd0 tx=0x7fa7f0005850 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 msgr2=0x7fa7fc197920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec00c920 tx=0x7fa7ec00cde0 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 shutdown_connections 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7e40a2800 0x7fa7e40a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7fc1073a0 0x7fa7fc19eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7fc104a70 0x7fa7fc197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 --2- 172.21.15.23:0/765034438 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7fc102140 0x7fa7fc197920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 >> 172.21.15.23:0/765034438 conn(0x7fa7fc0fbb30 msgr2=0x7fa7fc102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:00.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 shutdown_connections 2024-03-28T15:50:00.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:00.052+0000 7fa801977640 1 -- 172.21.15.23:0/765034438 wait complete. 2024-03-28T15:50:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6832/2721990059,v1:172.21.15.117:6833/2721990059] up:boot 2024-03-28T15:50:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: fsmap 2 up:standby 2024-03-28T15:50:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:50:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: Deploying daemon mds.j on smithi023 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: overall HEALTH_OK 2024-03-28T15:50:00.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:00 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/765034438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6832/2721990059,v1:172.21.15.117:6833/2721990059] up:boot 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: fsmap 2 up:standby 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:00.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:00.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:00.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: Deploying daemon mds.j on smithi023 2024-03-28T15:50:00.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: overall HEALTH_OK 2024-03-28T15:50:00.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:00 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/765034438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6832/2721990059,v1:172.21.15.117:6833/2721990059] up:boot 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: fsmap 2 up:standby 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: Deploying daemon mds.j on smithi023 2024-03-28T15:50:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: overall HEALTH_OK 2024-03-28T15:50:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:00 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/765034438' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:01.781 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:01 smithi023 ceph-mon[32448]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:01.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:01 smithi117 ceph-mon[36002]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:01.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:01 smithi120 ceph-mon[36826]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:02.066 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:02.093 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:01 smithi023 systemd[1]: Starting Ceph mds.j for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 -- 172.21.15.23:0/3683626482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c102140 msgr2=0x7fd56c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 --2- 172.21.15.23:0/3683626482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c102140 0x7fd56c104530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd560009900 tx=0x7fd56002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 -- 172.21.15.23:0/3683626482 shutdown_connections 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 --2- 172.21.15.23:0/3683626482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 --2- 172.21.15.23:0/3683626482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c104a70 0x7fd56c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.173+0000 7fd571c27640 1 --2- 172.21.15.23:0/3683626482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c102140 0x7fd56c104530 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.174+0000 7fd571c27640 1 -- 172.21.15.23:0/3683626482 >> 172.21.15.23:0/3683626482 conn(0x7fd56c0fbb30 msgr2=0x7fd56c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.174+0000 7fd571c27640 1 -- 172.21.15.23:0/3683626482 shutdown_connections 2024-03-28T15:50:02.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.174+0000 7fd571c27640 1 -- 172.21.15.23:0/3683626482 wait complete. 2024-03-28T15:50:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.174+0000 7fd571c27640 1 Processor -- start 2024-03-28T15:50:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.174+0000 7fd571c27640 1 -- start start 2024-03-28T15:50:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 0x7fd56c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:02.178 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c19efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10d350 con 0x7fd56c104a70 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10d1d0 con 0x7fd56c1073a0 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10d4d0 con 0x7fd56c102140 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54800/0 (socket says 172.21.15.23:54800) 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 -- 172.21.15.23:0/2671759212 learned_addr learned my addr 172.21.15.23:0/2671759212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56bfff640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c19efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56b7fe640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 0x7fd56c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 -- 172.21.15.23:0/2671759212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 msgr2=0x7fd56c197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 0x7fd56c197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 -- 172.21.15.23:0/2671759212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 msgr2=0x7fd56c19efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:02.179 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c19efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 -- 172.21.15.23:0/2671759212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c19f650 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56bfff640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56b7fe640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 0x7fd56c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd56affd640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd55c005fd0 tx=0x7fd55c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c017070 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd55c009c30 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c19f940 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c010460 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.175+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c19fe90 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.176+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c0fd620 con 0x7fd56c104a70 2024-03-28T15:50:02.180 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.177+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd55c003e70 con 0x7fd56c104a70 2024-03-28T15:50:02.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.178+0000 7fd568ff9640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 0x7fd5540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:02.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.178+0000 7fd56b7fe640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 0x7fd5540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:02.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.178+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd55c0ce3a0 con 0x7fd56c104a70 2024-03-28T15:50:02.181 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.178+0000 7fd56b7fe640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 0x7fd5540a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd5600096c0 tx=0x7fd560000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:02.183 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.182+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd55c020030 con 0x7fd56c104a70 2024-03-28T15:50:02.356 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 podman[72191]: 2024-03-28 15:50:02.078722038 +0000 UTC m=+0.018444232 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:02.356 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 podman[72191]: 2024-03-28 15:50:02.245514934 +0000 UTC m=+0.185237107 container create 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T15:50:02.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.410+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd56c02d2c0 con 0x7fd56c104a70 2024-03-28T15:50:02.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.411+0000 7fd568ff9640 1 -- 172.21.15.23:0/2671759212 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2119 (secure 0 0 0) 0x7fd55c09b8e0 con 0x7fd56c104a70 2024-03-28T15:50:02.412 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:02.412 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:50:02.413 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 3 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 msgr2=0x7fd5540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 0x7fd5540a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd5600096c0 tx=0x7fd560000bc0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 msgr2=0x7fd56c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd55c005fd0 tx=0x7fd55c00dde0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 shutdown_connections 2024-03-28T15:50:02.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd5540a2800 0x7fd5540a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd56c1073a0 0x7fd56c19efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd56c104a70 0x7fd56c197f70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 --2- 172.21.15.23:0/2671759212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd56c102140 0x7fd56c197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 >> 172.21.15.23:0/2671759212 conn(0x7fd56c0fbb30 msgr2=0x7fd56c102b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 shutdown_connections 2024-03-28T15:50:02.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:02.413+0000 7fd571c27640 1 -- 172.21.15.23:0/2671759212 wait complete. 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:02 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2671759212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 podman[72191]: 2024-03-28 15:50:02.456975819 +0000 UTC m=+0.396697994 container init 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 podman[72191]: 2024-03-28 15:50:02.459652591 +0000 UTC m=+0.399374756 container start 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 ceph-mds[72251]: -- 172.21.15.23:0/1831881614 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56419fbcba40 con 0x56419fb9e800 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j[72245]: starting mds.j at 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 bash[72191]: 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad 2024-03-28T15:50:02.671 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 15:50:02 smithi023 systemd[1]: Started Ceph mds.j for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:02 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2671759212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:02 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2671759212' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6834/1890623637,v1:172.21.15.23:6835/1890623637] up:boot 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: fsmap 3 up:standby 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:03 smithi117 ceph-mon[36002]: Deploying daemon mds.d on smithi023 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6834/1890623637,v1:172.21.15.23:6835/1890623637] up:boot 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: fsmap 3 up:standby 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:03.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:03 smithi120 ceph-mon[36826]: Deploying daemon mds.d on smithi023 2024-03-28T15:50:03.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6834/1890623637,v1:172.21.15.23:6835/1890623637] up:boot 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: fsmap 3 up:standby 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:03.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:03 smithi023 ceph-mon[32448]: Deploying daemon mds.d on smithi023 2024-03-28T15:50:04.425 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 -- 172.21.15.23:0/640125404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab0103340 msgr2=0x7feab0109bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 --2- 172.21.15.23:0/640125404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab0103340 0x7feab0109bd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7feaa8009cd0 tx=0x7feaa802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 -- 172.21.15.23:0/640125404 shutdown_connections 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 --2- 172.21.15.23:0/640125404 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab0103340 0x7feab0109bd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 --2- 172.21.15.23:0/640125404 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0102980 0x7feab0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 --2- 172.21.15.23:0/640125404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab00ff9b0 0x7feab00ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 -- 172.21.15.23:0/640125404 >> 172.21.15.23:0/640125404 conn(0x7feab00fb2e0 msgr2=0x7feab00fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 -- 172.21.15.23:0/640125404 shutdown_connections 2024-03-28T15:50:04.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.678+0000 7feab4f89640 1 -- 172.21.15.23:0/640125404 wait complete. 2024-03-28T15:50:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 Processor -- start 2024-03-28T15:50:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 -- start start 2024-03-28T15:50:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:04.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 0x7feab019a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 0x7feab01a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab010d390 con 0x7feab00ff9b0 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab010d210 con 0x7feab0102980 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feab4f89640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab010d510 con 0x7feab0103340 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.679+0000 7feaae59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54850/0 (socket says 172.21.15.23:54850) 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 -- 172.21.15.23:0/3390001782 learned_addr learned my addr 172.21.15.23:0/3390001782 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaadd9b640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 0x7feab019a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaaed9d640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 0x7feab01a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 -- 172.21.15.23:0/3390001782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 msgr2=0x7feab01a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 0x7feab01a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 -- 172.21.15.23:0/3390001782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 msgr2=0x7feab019a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:04.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 0x7feab019a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 -- 172.21.15.23:0/3390001782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab01a17f0 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaaed9d640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 0x7feab01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaadd9b640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 0x7feab019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feaae59c640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feaa000d950 tx=0x7feaa000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa0017020 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab01a1ae0 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaa0009c30 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa000f6b0 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.680+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab01a2080 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.681+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab01a22b0 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.682+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feaa001a020 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.683+0000 7fea9f7fe640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 0x7fea980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.683+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feaa00ce6c0 con 0x7feab00ff9b0 2024-03-28T15:50:04.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.683+0000 7feaadd9b640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 0x7fea980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:04.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.683+0000 7feaadd9b640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 0x7fea980a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feaa4005d70 tx=0x7feaa4005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:04.692 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.692+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feaa0021020 con 0x7feab00ff9b0 2024-03-28T15:50:04.921 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:04 smithi023 systemd[1]: Starting Ceph mds.d for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:04.925 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.924+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feab01a2480 con 0x7feab00ff9b0 2024-03-28T15:50:04.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.925+0000 7fea9f7fe640 1 -- 172.21.15.23:0/3390001782 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2910 (secure 0 0 0) 0x7feaa009bc00 con 0x7feab00ff9b0 2024-03-28T15:50:04.926 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:04.926 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:50:04.927 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 4 2024-03-28T15:50:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 msgr2=0x7fea980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:04.928 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 0x7fea980a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7feaa4005d70 tx=0x7feaa4005d00 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 msgr2=0x7feab0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feaa000d950 tx=0x7feaa000de10 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 shutdown_connections 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fea980a2800 0x7fea980a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7feab0103340 0x7feab01a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab0102980 0x7feab019a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 --2- 172.21.15.23:0/3390001782 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7feab00ff9b0 0x7feab0199bd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 >> 172.21.15.23:0/3390001782 conn(0x7feab00fb2e0 msgr2=0x7feab0100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 shutdown_connections 2024-03-28T15:50:04.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:04.927+0000 7feab4f89640 1 -- 172.21.15.23:0/3390001782 wait complete. 2024-03-28T15:50:05.198 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 podman[72573]: 2024-03-28 15:50:04.91922287 +0000 UTC m=+0.017869417 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:05.198 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 podman[72573]: 2024-03-28 15:50:05.060206765 +0000 UTC m=+0.158853300 container create e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-28T15:50:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3390001782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 podman[72573]: 2024-03-28 15:50:05.312966099 +0000 UTC m=+0.411612637 container init e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 podman[72573]: 2024-03-28 15:50:05.315476546 +0000 UTC m=+0.414123082 container start e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-mds[72595]: -- 172.21.15.23:0/590949339 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558338d0da40 con 0x558338ce1000 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d[72591]: starting mds.d at 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 bash[72573]: e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 2024-03-28T15:50:05.671 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 15:50:05 smithi023 systemd[1]: Started Ceph mds.d for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3390001782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:05.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:05 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3390001782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:05 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:06 smithi117 ceph-mon[36002]: Deploying daemon mds.e on smithi117 2024-03-28T15:50:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:06 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6836/252463346,v1:172.21.15.23:6837/252463346] up:boot 2024-03-28T15:50:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:06 smithi117 ceph-mon[36002]: fsmap 4 up:standby 2024-03-28T15:50:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:06 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:50:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:06 smithi120 ceph-mon[36826]: Deploying daemon mds.e on smithi117 2024-03-28T15:50:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:06 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6836/252463346,v1:172.21.15.23:6837/252463346] up:boot 2024-03-28T15:50:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:06 smithi120 ceph-mon[36826]: fsmap 4 up:standby 2024-03-28T15:50:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:06 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:50:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:06 smithi023 ceph-mon[32448]: Deploying daemon mds.e on smithi117 2024-03-28T15:50:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:06 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6836/252463346,v1:172.21.15.23:6837/252463346] up:boot 2024-03-28T15:50:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:06 smithi023 ceph-mon[32448]: fsmap 4 up:standby 2024-03-28T15:50:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:06 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:50:06.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.056+0000 7efcc2b2f640 1 -- 172.21.15.23:0/1325971937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc100640 msgr2=0x7efcbc100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.056+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/1325971937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc100640 0x7efcbc100a60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009900 tx=0x7efcb002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- 172.21.15.23:0/1325971937 shutdown_connections 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/1325971937 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc1021f0 0x7efcbc108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/1325971937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 0x7efcbc101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/1325971937 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc100640 0x7efcbc100a60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- 172.21.15.23:0/1325971937 >> 172.21.15.23:0/1325971937 conn(0x7efcbc0fbbd0 msgr2=0x7efcbc0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- 172.21.15.23:0/1325971937 shutdown_connections 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- 172.21.15.23:0/1325971937 wait complete. 2024-03-28T15:50:07.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 Processor -- start 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- start start 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc100640 0x7efcbc199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 0x7efcbc199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc10c240 con 0x7efcbc1021f0 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc10c0c0 con 0x7efcbc101830 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.057+0000 7efcc2b2f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc10c3c0 con 0x7efcbc100640 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcbbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 0x7efcbc199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54894/0 (socket says 172.21.15.23:54894) 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 -- 172.21.15.23:0/787694339 learned_addr learned my addr 172.21.15.23:0/787694339 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc08cb640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc100640 0x7efcbc199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 -- 172.21.15.23:0/787694339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc100640 msgr2=0x7efcbc199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:07.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc100640 0x7efcbc199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 -- 172.21.15.23:0/787694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 msgr2=0x7efcbc199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 0x7efcbc199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 -- 172.21.15.23:0/787694339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc1a1590 con 0x7efcbc1021f0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc10cc640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcb400eaa0 tx=0x7efcb400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb4004d20 con 0x7efcbc1021f0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcb4005020 con 0x7efcbc1021f0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1a1880 con 0x7efcbc1021f0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb40116d0 con 0x7efcbc1021f0 2024-03-28T15:50:07.062 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.058+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc1a1dd0 con 0x7efcbc1021f0 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.059+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcbc106170 con 0x7efcbc1021f0 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.060+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efcb40051d0 con 0x7efcbc1021f0 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.061+0000 7efcb9ffb640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 0x7efcac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.061+0000 7efcc08cb640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 0x7efcac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.061+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efcb40ce7f0 con 0x7efcbc1021f0 2024-03-28T15:50:07.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.062+0000 7efcc08cb640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 0x7efcac0a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009230 tx=0x7efcb0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:07.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.066+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efcb4016030 con 0x7efcbc1021f0 2024-03-28T15:50:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.289+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcbc0622d0 con 0x7efcbc1021f0 2024-03-28T15:50:07.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.289+0000 7efcb9ffb640 1 -- 172.21.15.23:0/787694339 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3698 (secure 0 0 0) 0x7efcb409bd30 con 0x7efcbc1021f0 2024-03-28T15:50:07.291 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:07.291 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-28T15:50:07.291 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 5 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.291+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 msgr2=0x7efcac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.291+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 0x7efcac0a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7efcb0009230 tx=0x7efcb0000bc0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.291+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 msgr2=0x7efcbc1a0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.291+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7efcb400eaa0 tx=0x7efcb400eea0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 shutdown_connections 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efcac0a2800 0x7efcac0a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efcbc1021f0 0x7efcbc1a0f20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcbc101830 0x7efcbc199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 --2- 172.21.15.23:0/787694339 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efcbc100640 0x7efcbc199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:07.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 >> 172.21.15.23:0/787694339 conn(0x7efcbc0fbbd0 msgr2=0x7efcbc06a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:07.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 shutdown_connections 2024-03-28T15:50:07.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:07.292+0000 7efcc2b2f640 1 -- 172.21.15.23:0/787694339 wait complete. 2024-03-28T15:50:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:07 smithi023 ceph-mon[32448]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:07.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:07 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/787694339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:07.803 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:07 smithi117 ceph-mon[36002]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:07.803 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:07 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/787694339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:07.803 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:07 smithi117 systemd[1]: Starting Ceph mds.e for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:07 smithi120 ceph-mon[36826]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:07 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/787694339' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:08.064 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:07 smithi117 podman[65682]: 2024-03-28 15:50:07.705768936 +0000 UTC m=+0.017737431 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:08.065 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:07 smithi117 podman[65682]: 2024-03-28 15:50:07.821606031 +0000 UTC m=+0.133574507 container create 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-28T15:50:08.065 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 podman[65682]: 2024-03-28 15:50:08.065041628 +0000 UTC m=+0.377010112 container init 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-28T15:50:08.321 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 podman[65682]: 2024-03-28 15:50:08.067854199 +0000 UTC m=+0.379822685 container start 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T15:50:08.321 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 ceph-mds[65701]: -- 172.21.15.117:0/568714628 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d1a3247a40 con 0x55d1a321ac00 2024-03-28T15:50:08.321 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: starting mds.e at 2024-03-28T15:50:08.322 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 bash[65682]: 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 2024-03-28T15:50:08.322 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 15:50:08 smithi117 systemd[1]: Started Ceph mds.e for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:09.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.418+0000 7fbb06ee8640 1 -- 172.21.15.23:0/1043402227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb001021f0 msgr2=0x7fbb00108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.418+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/1043402227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb001021f0 0x7fbb00108a80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009ca0 tx=0x7fbaf802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- 172.21.15.23:0/1043402227 shutdown_connections 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/1043402227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb001021f0 0x7fbb00108a80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/1043402227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00101830 0x7fbb00101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/1043402227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb00100640 0x7fbb00100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- 172.21.15.23:0/1043402227 >> 172.21.15.23:0/1043402227 conn(0x7fbb000fbbd0 msgr2=0x7fbb000fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- 172.21.15.23:0/1043402227 shutdown_connections 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- 172.21.15.23:0/1043402227 wait complete. 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 Processor -- start 2024-03-28T15:50:09.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- start start 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00100640 0x7fbb001978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb001021f0 0x7fbb0019ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0010c230 con 0x7fbb00101830 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0010c0b0 con 0x7fbb00100640 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.419+0000 7fbb06ee8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0010c3b0 con 0x7fbb001021f0 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50220/0 (socket says 172.21.15.23:50220) 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 -- 172.21.15.23:0/2958627446 learned_addr learned my addr 172.21.15.23:0/2958627446 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbb04c84640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00100640 0x7fbb001978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbb05485640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb001021f0 0x7fbb0019ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 -- 172.21.15.23:0/2958627446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb001021f0 msgr2=0x7fbb0019ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb001021f0 0x7fbb0019ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 -- 172.21.15.23:0/2958627446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00100640 msgr2=0x7fbb001978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:09.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00100640 0x7fbb001978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 -- 172.21.15.23:0/2958627446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb0019f500 con 0x7fbb00101830 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafffff640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00d950 tx=0x7fbaec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec017070 con 0x7fbb00101830 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0019f7f0 con 0x7fbb00101830 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaec009c30 con 0x7fbb00101830 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaec00f6b0 con 0x7fbb00101830 2024-03-28T15:50:09.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.420+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0019fd40 con 0x7fbb00101830 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.421+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0019ffa0 con 0x7fbb00101830 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.422+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbaec0109d0 con 0x7fbb00101830 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.423+0000 7fbafdffb640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 0x7fbaf00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.423+0000 7fbb04c84640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 0x7fbaf00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.423+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbaec0ce270 con 0x7fbb00101830 2024-03-28T15:50:09.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.424+0000 7fbb04c84640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 0x7fbaf00a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb00101690 tx=0x7fbaf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:09.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.428+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbaec009de0 con 0x7fbb00101830 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: Deploying daemon mds.f on smithi120 2024-03-28T15:50:09.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6834/3656482331,v1:172.21.15.117:6835/3656482331] up:boot 2024-03-28T15:50:09.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: fsmap 5 up:standby 2024-03-28T15:50:09.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:09 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:09.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:09.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: Deploying daemon mds.f on smithi120 2024-03-28T15:50:09.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6834/3656482331,v1:172.21.15.117:6835/3656482331] up:boot 2024-03-28T15:50:09.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: fsmap 5 up:standby 2024-03-28T15:50:09.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:09 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:50:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.649+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb001a0670 con 0x7fbb00101830 2024-03-28T15:50:09.650 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.649+0000 7fbafdffb640 1 -- 172.21.15.23:0/2958627446 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4492 (secure 0 0 0) 0x7fbaec09b7b0 con 0x7fbb00101830 2024-03-28T15:50:09.651 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:09.651 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-28T15:50:09.651 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 6 2024-03-28T15:50:09.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 msgr2=0x7fbaf00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 0x7fbaf00a4cc0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fbb00101690 tx=0x7fbaf4005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 msgr2=0x7fbb00197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00d950 tx=0x7fbaec00de10 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 shutdown_connections 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbaf00a2800 0x7fbaf00a4cc0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbb001021f0 0x7fbb0019ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbb00101830 0x7fbb00197e20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 --2- 172.21.15.23:0/2958627446 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb00100640 0x7fbb001978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 >> 172.21.15.23:0/2958627446 conn(0x7fbb000fbbd0 msgr2=0x7fbb000fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 shutdown_connections 2024-03-28T15:50:09.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:09.651+0000 7fbb06ee8640 1 -- 172.21.15.23:0/2958627446 wait complete. 2024-03-28T15:50:09.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:09.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:09.662 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:09.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:09.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: Deploying daemon mds.f on smithi120 2024-03-28T15:50:09.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6834/3656482331,v1:172.21.15.117:6835/3656482331] up:boot 2024-03-28T15:50:09.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: fsmap 5 up:standby 2024-03-28T15:50:09.663 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:09 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:50:10.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:10 smithi117 ceph-mon[36002]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:10.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:10 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2958627446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:10.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:10 smithi120 ceph-mon[36826]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:10.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:10 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2958627446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:10.589 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 systemd[1]: Starting Ceph mds.f for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:10 smithi023 ceph-mon[32448]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:10 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2958627446' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:10.952 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 podman[66464]: 2024-03-28 15:50:10.563034585 +0000 UTC m=+0.017386408 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:10.952 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 podman[66464]: 2024-03-28 15:50:10.682408994 +0000 UTC m=+0.136760868 container create 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T15:50:10.952 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 podman[66464]: 2024-03-28 15:50:10.951945501 +0000 UTC m=+0.406297337 container init 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-28T15:50:11.339 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 podman[66464]: 2024-03-28 15:50:10.954540427 +0000 UTC m=+0.408892260 container start 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T15:50:11.339 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 ceph-mds[66483]: -- 172.21.15.120:0/2847992700 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55eac204ba40 con 0x55eac201ec00 2024-03-28T15:50:11.339 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: starting mds.f at 2024-03-28T15:50:11.339 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:11 smithi120 bash[66464]: 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc 2024-03-28T15:50:11.339 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 15:50:11 smithi120 systemd[1]: Started Ceph mds.f for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:11.664 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 -- 172.21.15.23:0/2546794178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c0fffb0 msgr2=0x7f8a5c1003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 --2- 172.21.15.23:0/2546794178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c1003d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8a58009900 tx=0x7f8a5802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 -- 172.21.15.23:0/2546794178 shutdown_connections 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 --2- 172.21.15.23:0/2546794178 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 --2- 172.21.15.23:0/2546794178 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c100910 0x7f8a5c100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 --2- 172.21.15.23:0/2546794178 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c1003d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 -- 172.21.15.23:0/2546794178 >> 172.21.15.23:0/2546794178 conn(0x7f8a5c0fbbd0 msgr2=0x7f8a5c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 -- 172.21.15.23:0/2546794178 shutdown_connections 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.777+0000 7f8a6369a640 1 -- 172.21.15.23:0/2546794178 wait complete. 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 Processor -- start 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 -- start start 2024-03-28T15:50:11.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c100910 0x7f8a5c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10f510 con 0x7f8a5c1012b0 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10f390 con 0x7f8a5c0fffb0 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a6369a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10f690 con 0x7f8a5c100910 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50246/0 (socket says 172.21.15.23:50246) 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 -- 172.21.15.23:0/3980683694 learned_addr learned my addr 172.21.15.23:0/3980683694 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a60c35640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c100910 0x7f8a5c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61436640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 -- 172.21.15.23:0/3980683694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c100910 msgr2=0x7f8a5c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c100910 0x7f8a5c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 -- 172.21.15.23:0/3980683694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c0fffb0 msgr2=0x7f8a5c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.778+0000 7f8a61c37640 1 -- 172.21.15.23:0/3980683694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c1a1880 con 0x7f8a5c1012b0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a61c37640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400efd0 tx=0x7f8a5400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54003b70 con 0x7f8a5c1012b0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a54003d10 con 0x7f8a5c1012b0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c1a1b70 con 0x7f8a5c1012b0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a540109f0 con 0x7f8a5c1012b0 2024-03-28T15:50:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.779+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c1a2070 con 0x7f8a5c1012b0 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.780+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c10ad00 con 0x7f8a5c1012b0 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.781+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8a54004020 con 0x7f8a5c1012b0 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.781+0000 7f8a527fc640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 0x7f8a480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.782+0000 7f8a61436640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 0x7f8a480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.782+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8a540ceb70 con 0x7f8a5c1012b0 2024-03-28T15:50:11.784 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.782+0000 7f8a61436640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 0x7f8a480a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006010 tx=0x7f8a5802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:11.786+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a5409c0b0 con 0x7f8a5c1012b0 2024-03-28T15:50:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.007+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a5c02d070 con 0x7f8a5c1012b0 2024-03-28T15:50:12.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.007+0000 7f8a527fc640 1 -- 172.21.15.23:0/3980683694 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5286 (secure 0 0 0) 0x7f8a5409b9b0 con 0x7f8a5c1012b0 2024-03-28T15:50:12.008 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:12.009 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-28T15:50:12.009 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 7 2024-03-28T15:50:12.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 msgr2=0x7f8a480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:12.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 0x7f8a480a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a58006010 tx=0x7f8a5802c8a0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.010 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 msgr2=0x7f8a5c1a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8a5400efd0 tx=0x7f8a5400c6a0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 shutdown_connections 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a480a2800 0x7f8a480a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1012b0 0x7f8a5c1a11e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c100910 0x7f8a5c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 --2- 172.21.15.23:0/3980683694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c0fffb0 0x7f8a5c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 >> 172.21.15.23:0/3980683694 conn(0x7f8a5c0fbbd0 msgr2=0x7f8a5c0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 shutdown_connections 2024-03-28T15:50:12.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:12.009+0000 7f8a6369a640 1 -- 172.21.15.23:0/3980683694 wait complete. 2024-03-28T15:50:12.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:12.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: Deploying daemon mds.c on smithi120 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6834/1615812668,v1:172.21.15.120:6835/1615812668] up:boot 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: fsmap 6 up:standby 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:50:12.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:12 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3980683694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: Deploying daemon mds.c on smithi120 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6834/1615812668,v1:172.21.15.120:6835/1615812668] up:boot 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: fsmap 6 up:standby 2024-03-28T15:50:12.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:50:12.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:12 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3980683694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: Deploying daemon mds.c on smithi120 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6834/1615812668,v1:172.21.15.120:6835/1615812668] up:boot 2024-03-28T15:50:12.453 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: fsmap 6 up:standby 2024-03-28T15:50:12.454 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:50:12.454 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:12 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3980683694' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:13.339 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 systemd[1]: Starting Ceph mds.c for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:13.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 podman[66769]: 2024-03-28 15:50:13.372139024 +0000 UTC m=+0.017789938 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:13.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 podman[66769]: 2024-03-28 15:50:13.489489265 +0000 UTC m=+0.135140166 container create 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-28T15:50:13.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 podman[66769]: 2024-03-28 15:50:13.742515848 +0000 UTC m=+0.388166758 container init 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-28T15:50:14.022 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:14.089 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 podman[66769]: 2024-03-28 15:50:13.74504017 +0000 UTC m=+0.390691064 container start 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T15:50:14.089 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 ceph-mds[66788]: -- 172.21.15.120:0/4068815490 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b21a3a5a40 con 0x55b21a378c00 2024-03-28T15:50:14.089 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: starting mds.c at 2024-03-28T15:50:14.089 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 bash[66769]: 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e 2024-03-28T15:50:14.089 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 15:50:13 smithi120 systemd[1]: Started Ceph mds.c for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:14.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 -- 172.21.15.23:0/1423843594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 msgr2=0x7f46b8100a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/1423843594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8100a60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009900 tx=0x7f46ac02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 -- 172.21.15.23:0/1423843594 shutdown_connections 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/1423843594 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b81021f0 0x7f46b8108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/1423843594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b8101830 0x7f46b8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/1423843594 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8100a60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 -- 172.21.15.23:0/1423843594 >> 172.21.15.23:0/1423843594 conn(0x7f46b80fbbd0 msgr2=0x7f46b80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 -- 172.21.15.23:0/1423843594 shutdown_connections 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.137+0000 7f46bf1f4640 1 -- 172.21.15.23:0/1423843594 wait complete. 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 Processor -- start 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 -- start start 2024-03-28T15:50:14.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b8101830 0x7f46b8197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b81021f0 0x7f46b819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810c240 con 0x7f46b8100640 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810c0c0 con 0x7f46b81021f0 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bf1f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b810c3c0 con 0x7f46b8101830 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bcf90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bcf90640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50260/0 (socket says 172.21.15.23:50260) 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bcf90640 1 -- 172.21.15.23:0/2241223152 learned_addr learned my addr 172.21.15.23:0/2241223152 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46bd791640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b81021f0 0x7f46b819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.138+0000 7f46b7fff640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b8101830 0x7f46b8197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 -- 172.21.15.23:0/2241223152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b8101830 msgr2=0x7f46b8197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b8101830 0x7f46b8197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 -- 172.21.15.23:0/2241223152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b81021f0 msgr2=0x7f46b819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b81021f0 0x7f46b819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 -- 172.21.15.23:0/2241223152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b819f630 con 0x7f46b8100640 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bcf90640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f46ac000c00 tx=0x7f46ac02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac041070 con 0x7f46b8100640 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46ac004450 con 0x7f46b8100640 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b819f8c0 con 0x7f46b8100640 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ac034440 con 0x7f46b8100640 2024-03-28T15:50:14.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.139+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b819fdb0 con 0x7f46b8100640 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.140+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b819ffb0 con 0x7f46b8100640 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.141+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f46ac044070 con 0x7f46b8100640 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.142+0000 7f46b5ffb640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.142+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f46ac0f0080 con 0x7f46b8100640 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.142+0000 7f46b7fff640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:14.144 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.142+0000 7f46b7fff640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 0x7f46a80a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46b8198f20 tx=0x7f46a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:14.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.146+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46ac0bd000 con 0x7f46b8100640 2024-03-28T15:50:14.371 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.370+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46b81a0180 con 0x7f46b8100640 2024-03-28T15:50:14.372 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.371+0000 7f46b5ffb640 1 -- 172.21.15.23:0/2241223152 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7f46ac0c10a0 con 0x7f46b8100640 2024-03-28T15:50:14.372 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:14.372 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T15:50:14.372 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 msgr2=0x7f46a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 0x7f46a80a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f46b8198f20 tx=0x7f46a4005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 msgr2=0x7f46b8197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f46ac000c00 tx=0x7f46ac02c640 comp rx=0 tx=0).stop 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 shutdown_connections 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46a80a2800 0x7f46a80a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46b81021f0 0x7f46b819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46b8101830 0x7f46b8197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.374 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 --2- 172.21.15.23:0/2241223152 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46b8100640 0x7f46b8197a60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:14.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 >> 172.21.15.23:0/2241223152 conn(0x7f46b80fbbd0 msgr2=0x7f46b8069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:14.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 shutdown_connections 2024-03-28T15:50:14.375 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:14.373+0000 7f46bf1f4640 1 -- 172.21.15.23:0/2241223152 wait complete. 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:14.384 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:14 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:14.545 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:14 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:14 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:15.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:15 smithi023 ceph-mon[32448]: Deploying daemon mds.h on smithi117 2024-03-28T15:50:15.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:15 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:boot 2024-03-28T15:50:15.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:15 smithi023 ceph-mon[32448]: fsmap 7 up:standby 2024-03-28T15:50:15.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:15 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:50:15.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:15 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2241223152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:15 smithi117 ceph-mon[36002]: Deploying daemon mds.h on smithi117 2024-03-28T15:50:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:15 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:boot 2024-03-28T15:50:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:15 smithi117 ceph-mon[36002]: fsmap 7 up:standby 2024-03-28T15:50:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:15 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:50:15.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:15 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2241223152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:15.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:15 smithi120 ceph-mon[36826]: Deploying daemon mds.h on smithi117 2024-03-28T15:50:15.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:15 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:boot 2024-03-28T15:50:15.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:15 smithi120 ceph-mon[36826]: fsmap 7 up:standby 2024-03-28T15:50:15.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:15 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:50:15.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:15 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2241223152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:16 smithi117 ceph-mon[36002]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:16.321 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 systemd[1]: Starting Ceph mds.h for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:16.386 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:16.415 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:16 smithi023 ceph-mon[32448]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:16.490 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:16 smithi120 ceph-mon[36826]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/1534205071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e841021c0 msgr2=0x7f0e841045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/1534205071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e841021c0 0x7f0e841045b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009900 tx=0x7f0e8002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/1534205071 shutdown_connections 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/1534205071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e84107420 0x7f0e84109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/1534205071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 0x7f0e84106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/1534205071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e841021c0 0x7f0e841045b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/1534205071 >> 172.21.15.23:0/1534205071 conn(0x7f0e840fbbd0 msgr2=0x7f0e840fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:16.504 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/1534205071 shutdown_connections 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/1534205071 wait complete. 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 Processor -- start 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.501+0000 7f0e8c4dd640 1 -- start start 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e841021c0 0x7f0e841979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 0x7f0e84197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8410d370 con 0x7f0e84107420 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8410d1f0 con 0x7f0e841021c0 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8410d4f0 con 0x7f0e84104af0 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50284/0 (socket says 172.21.15.23:50284) 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 -- 172.21.15.23:0/3644218713 learned_addr learned my addr 172.21.15.23:0/3644218713 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:16.505 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e89a78640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 0x7f0e84197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8a279640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e841021c0 0x7f0e841979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 -- 172.21.15.23:0/3644218713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 msgr2=0x7f0e84197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 0x7f0e84197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 -- 172.21.15.23:0/3644218713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e841021c0 msgr2=0x7f0e841979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e841021c0 0x7f0e841979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 -- 172.21.15.23:0/3644218713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e8419f5d0 con 0x7f0e84107420 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8aa7a640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00cce0 tx=0x7f0e7c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c00f450 con 0x7f0e84107420 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e7c00fad0 con 0x7f0e84107420 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8419f8c0 con 0x7f0e84107420 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c018c10 con 0x7f0e84107420 2024-03-28T15:50:16.506 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.502+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8419fdc0 con 0x7f0e84107420 2024-03-28T15:50:16.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.503+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e840fd6c0 con 0x7f0e84107420 2024-03-28T15:50:16.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.505+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0e7c00f5f0 con 0x7f0e84107420 2024-03-28T15:50:16.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.505+0000 7f0e7b7fe640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 0x7f0e700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:16.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.505+0000 7f0e8a279640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 0x7f0e700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:16.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.505+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0e7c0ced00 con 0x7f0e84107420 2024-03-28T15:50:16.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.506+0000 7f0e8a279640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 0x7f0e700a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e80006010 tx=0x7f0e80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:16.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.509+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0e7c014030 con 0x7f0e84107420 2024-03-28T15:50:16.661 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 podman[65993]: 2024-03-28 15:50:16.288753903 +0000 UTC m=+0.017472529 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:16.662 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 podman[65993]: 2024-03-28 15:50:16.39265385 +0000 UTC m=+0.121372487 container create 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T15:50:16.662 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 podman[65993]: 2024-03-28 15:50:16.661805337 +0000 UTC m=+0.390523946 container init 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T15:50:16.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.732+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e8402d070 con 0x7f0e84107420 2024-03-28T15:50:16.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.733+0000 7f0e7b7fe640 1 -- 172.21.15.23:0/3644218713 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6080 (secure 0 0 0) 0x7f0e7c09c2d0 con 0x7f0e84107420 2024-03-28T15:50:16.734 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:16.734 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-28T15:50:16.734 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 8 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 msgr2=0x7f0e700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 0x7f0e700a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0e80006010 tx=0x7f0e80000bc0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 msgr2=0x7f0e8419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c00cce0 tx=0x7f0e7c00a620 comp rx=0 tx=0).stop 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 shutdown_connections 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0e700a2800 0x7f0e700a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0e84107420 0x7f0e8419ef60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0e84104af0 0x7f0e84197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 --2- 172.21.15.23:0/3644218713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0e841021c0 0x7f0e841979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:16.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 >> 172.21.15.23:0/3644218713 conn(0x7f0e840fbbd0 msgr2=0x7f0e84102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:16.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 shutdown_connections 2024-03-28T15:50:16.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:16.735+0000 7f0e8c4dd640 1 -- 172.21.15.23:0/3644218713 wait complete. 2024-03-28T15:50:17.071 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 podman[65993]: 2024-03-28 15:50:16.664433405 +0000 UTC m=+0.393152027 container start 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-28T15:50:17.071 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 ceph-mds[66012]: -- 172.21.15.117:0/1397423249 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b9654fba40 con 0x55b9654cf000 2024-03-28T15:50:17.071 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: starting mds.h at 2024-03-28T15:50:17.071 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 bash[65993]: 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 2024-03-28T15:50:17.071 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 15:50:16 smithi117 systemd[1]: Started Ceph mds.h for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:17.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3644218713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:17.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:17.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:17.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:17 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3644218713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:17.485 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:17 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3644218713' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:17.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:17 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:18 smithi120 ceph-mon[36826]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:18 smithi120 ceph-mon[36826]: Deploying daemon mds.i on smithi120 2024-03-28T15:50:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:18 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:boot 2024-03-28T15:50:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:18 smithi120 ceph-mon[36826]: fsmap 8 up:standby 2024-03-28T15:50:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:18 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:50:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:18 smithi023 ceph-mon[32448]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:18 smithi023 ceph-mon[32448]: Deploying daemon mds.i on smithi120 2024-03-28T15:50:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:18 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:boot 2024-03-28T15:50:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:18 smithi023 ceph-mon[32448]: fsmap 8 up:standby 2024-03-28T15:50:18.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:18 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:50:18.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:18 smithi117 ceph-mon[36002]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:18.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:18 smithi117 ceph-mon[36002]: Deploying daemon mds.i on smithi120 2024-03-28T15:50:18.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:18 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:boot 2024-03-28T15:50:18.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:18 smithi117 ceph-mon[36002]: fsmap 8 up:standby 2024-03-28T15:50:18.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:18 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:50:18.748 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:18.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 -- 172.21.15.23:0/2662831139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540107420 msgr2=0x7f2540109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:18.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 --2- 172.21.15.23:0/2662831139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540107420 0x7f2540109870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2538009900 tx=0x7f253802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 -- 172.21.15.23:0/2662831139 shutdown_connections 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 --2- 172.21.15.23:0/2662831139 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540107420 0x7f2540109870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 --2- 172.21.15.23:0/2662831139 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540104af0 0x7f2540106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 --2- 172.21.15.23:0/2662831139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 0x7f25401045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 -- 172.21.15.23:0/2662831139 >> 172.21.15.23:0/2662831139 conn(0x7f25400fbbd0 msgr2=0x7f25400fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 -- 172.21.15.23:0/2662831139 shutdown_connections 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.862+0000 7f2544eac640 1 -- 172.21.15.23:0/2662831139 wait complete. 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 Processor -- start 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 -- start start 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 0x7f2540197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:18.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540107420 0x7f254019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254010d370 con 0x7f2540104af0 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254010d1f0 con 0x7f25401021c0 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f2544eac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254010d4f0 con 0x7f2540107420 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39500/0 (socket says 172.21.15.23:39500) 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 -- 172.21.15.23:0/346579511 learned_addr learned my addr 172.21.15.23:0/346579511 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253e59c640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 0x7f2540197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253ed9d640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540107420 0x7f254019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 -- 172.21.15.23:0/346579511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540107420 msgr2=0x7f254019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540107420 0x7f254019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:18.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 -- 172.21.15.23:0/346579511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 msgr2=0x7f2540197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 0x7f2540197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.863+0000 7f253dd9b640 1 -- 172.21.15.23:0/346579511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254019f520 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f253dd9b640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f253400e9f0 tx=0x7f253400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253400ccc0 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2534005500 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254019f810 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2534010690 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.864+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254019fd10 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.865+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25400fd6c0 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.866+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f25340046c0 con 0x7f2540104af0 2024-03-28T15:50:18.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.866+0000 7f252f7fe640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 0x7f25280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:18.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.867+0000 7f253e59c640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 0x7f25280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:18.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.867+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25340ce920 con 0x7f2540104af0 2024-03-28T15:50:18.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.867+0000 7f253e59c640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 0x7f25280a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2530007ec0 tx=0x7f25300075b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:18.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:18.871+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f253409be60 con 0x7f2540104af0 2024-03-28T15:50:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.094+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254002d070 con 0x7f2540104af0 2024-03-28T15:50:19.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.094+0000 7f252f7fe640 1 -- 172.21.15.23:0/346579511 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6868 (secure 0 0 0) 0x7f253409b760 con 0x7f2540104af0 2024-03-28T15:50:19.096 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:19.097 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 9 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.096+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 msgr2=0x7f25280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.096+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 0x7f25280a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2530007ec0 tx=0x7f25300075b0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.096+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 msgr2=0x7f2540197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.096+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f253400e9f0 tx=0x7f253400eeb0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 shutdown_connections 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f25280a2800 0x7f25280a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2540107420 0x7f254019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2540104af0 0x7f2540197e40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 --2- 172.21.15.23:0/346579511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25401021c0 0x7f2540197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 >> 172.21.15.23:0/346579511 conn(0x7f25400fbbd0 msgr2=0x7f2540102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:19.100 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 shutdown_connections 2024-03-28T15:50:19.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:19.097+0000 7f2544eac640 1 -- 172.21.15.23:0/346579511 wait complete. 2024-03-28T15:50:19.284 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:19 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/346579511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:19.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:19 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/346579511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:19.489 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:19 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/346579511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:19.839 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 15:50:19 smithi120 systemd[1]: Started Ceph mds.i for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:20.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:20.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:20.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:20.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:20 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:20.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:20 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:20.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:20.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:20 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:21.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 -- 172.21.15.23:0/101507852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25501005c0 msgr2=0x7f25501009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:21.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 --2- 172.21.15.23:0/101507852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25501005c0 0x7f25501009e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2548009ca0 tx=0x7f254802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 -- 172.21.15.23:0/101507852 shutdown_connections 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 --2- 172.21.15.23:0/101507852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2550102170 0x7f2550108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 --2- 172.21.15.23:0/101507852 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25501017b0 0x7f2550101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 --2- 172.21.15.23:0/101507852 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f25501005c0 0x7f25501009e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 -- 172.21.15.23:0/101507852 >> 172.21.15.23:0/101507852 conn(0x7f25500fbb30 msgr2=0x7f25500fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 -- 172.21.15.23:0/101507852 shutdown_connections 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.215+0000 7f2555044640 1 -- 172.21.15.23:0/101507852 wait complete. 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 Processor -- start 2024-03-28T15:50:21.223 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 -- start start 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25501005c0 0x7f25501979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25501017b0 0x7f2550197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255010c210 con 0x7f2550102170 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.216+0000 7f2555044640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255010c090 con 0x7f25501017b0 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2555044640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255010c390 con 0x7f25501005c0 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:21.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39520/0 (socket says 172.21.15.23:39520) 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 -- 172.21.15.23:0/1416764818 learned_addr learned my addr 172.21.15.23:0/1416764818 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254e59c640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25501017b0 0x7f2550197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 -- 172.21.15.23:0/1416764818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25501005c0 msgr2=0x7f25501979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25501005c0 0x7f25501979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 -- 172.21.15.23:0/1416764818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25501017b0 msgr2=0x7f2550197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:21.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25501017b0 0x7f2550197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 -- 172.21.15.23:0/1416764818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255019f600 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f254f59e640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f254000d750 tx=0x7f254000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540017070 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f255019f8f0 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2540005500 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540004d90 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.217+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f255019fe40 con 0x7f2550102170 2024-03-28T15:50:21.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.218+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25501a00a0 con 0x7f2550102170 2024-03-28T15:50:21.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.219+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2540004f30 con 0x7f2550102170 2024-03-28T15:50:21.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.220+0000 7f2547fff640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 0x7f253c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:21.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.220+0000 7f254ed9d640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 0x7f253c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:21.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.220+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25400cdfa0 con 0x7f2550102170 2024-03-28T15:50:21.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.220+0000 7f254ed9d640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 0x7f253c0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2548006010 tx=0x7f25480093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:21.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.224+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f254009b4e0 con 0x7f2550102170 2024-03-28T15:50:21.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:21 smithi023 ceph-mon[32448]: Deploying daemon mds.g on smithi023 2024-03-28T15:50:21.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:21 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:boot 2024-03-28T15:50:21.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:21 smithi023 ceph-mon[32448]: fsmap 9 up:standby 2024-03-28T15:50:21.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:21 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:50:21.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.451+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25501a0430 con 0x7f2550102170 2024-03-28T15:50:21.452 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.451+0000 7f2547fff640 1 -- 172.21.15.23:0/1416764818 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7664 (secure 0 0 0) 0x7f254009ade0 con 0x7f2550102170 2024-03-28T15:50:21.453 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:21.453 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T15:50:21.453 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 10 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 msgr2=0x7f253c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 0x7f253c0a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2548006010 tx=0x7f25480093f0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 msgr2=0x7f255019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f254000d750 tx=0x7f254000dc10 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 shutdown_connections 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f253c0a2800 0x7f253c0a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2550102170 0x7f255019ef90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25501017b0 0x7f2550197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 --2- 172.21.15.23:0/1416764818 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f25501005c0 0x7f25501979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 >> 172.21.15.23:0/1416764818 conn(0x7f25500fbb30 msgr2=0x7f25500fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 shutdown_connections 2024-03-28T15:50:21.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:21.453+0000 7f2555044640 1 -- 172.21.15.23:0/1416764818 wait complete. 2024-03-28T15:50:21.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:21 smithi117 ceph-mon[36002]: Deploying daemon mds.g on smithi023 2024-03-28T15:50:21.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:21 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:boot 2024-03-28T15:50:21.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:21 smithi117 ceph-mon[36002]: fsmap 9 up:standby 2024-03-28T15:50:21.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:21 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:50:21.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:21 smithi120 ceph-mon[36826]: Deploying daemon mds.g on smithi023 2024-03-28T15:50:21.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:21 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:boot 2024-03-28T15:50:21.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:21 smithi120 ceph-mon[36826]: fsmap 9 up:standby 2024-03-28T15:50:21.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:21 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:50:22.142 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:21 smithi023 systemd[1]: Starting Ceph mds.g for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:22.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:22 smithi023 ceph-mon[32448]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:22.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:22 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1416764818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:22.420 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 podman[73171]: 2024-03-28 15:50:22.044699388 +0000 UTC m=+0.017909727 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:22.420 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 podman[73171]: 2024-03-28 15:50:22.15219019 +0000 UTC m=+0.125400528 container create be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-28T15:50:22.420 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 podman[73171]: 2024-03-28 15:50:22.363340134 +0000 UTC m=+0.336550475 container init be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD) 2024-03-28T15:50:22.421 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 podman[73171]: 2024-03-28 15:50:22.36637516 +0000 UTC m=+0.339585500 container start be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T15:50:22.421 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 ceph-mds[73190]: -- 172.21.15.23:0/4198550375 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca4c5e9a40 con 0x55ca4c5bd000 2024-03-28T15:50:22.421 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: starting mds.g at 2024-03-28T15:50:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:22 smithi117 ceph-mon[36002]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:22 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1416764818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:22 smithi120 ceph-mon[36826]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:22 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1416764818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:22.920 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 bash[73171]: be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 2024-03-28T15:50:22.920 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 15:50:22 smithi023 systemd[1]: Started Ceph mds.g for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:23.466 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.573+0000 7ffa48459640 1 -- 172.21.15.23:0/2704786094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa40102170 msgr2=0x7ffa40108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.573+0000 7ffa48459640 1 --2- 172.21.15.23:0/2704786094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa40102170 0x7ffa40108a00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ffa38009ca0 tx=0x7ffa3802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- 172.21.15.23:0/2704786094 shutdown_connections 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- 172.21.15.23:0/2704786094 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa40102170 0x7ffa40108a00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- 172.21.15.23:0/2704786094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401017b0 0x7ffa40101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- 172.21.15.23:0/2704786094 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa401005c0 0x7ffa401009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- 172.21.15.23:0/2704786094 >> 172.21.15.23:0/2704786094 conn(0x7ffa400fbb30 msgr2=0x7ffa400fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- 172.21.15.23:0/2704786094 shutdown_connections 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- 172.21.15.23:0/2704786094 wait complete. 2024-03-28T15:50:23.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 Processor -- start 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- start start 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401005c0 0x7ffa40197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa40102170 0x7ffa4019eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4010c220 con 0x7ffa401017b0 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4010c0a0 con 0x7ffa401005c0 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa48459640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4010c3a0 con 0x7ffa40102170 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa459f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa459f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39554/0 (socket says 172.21.15.23:39554) 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa459f4640 1 -- 172.21.15.23:0/1104461123 learned_addr learned my addr 172.21.15.23:0/1104461123 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.574+0000 7ffa461f5640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401005c0 0x7ffa40197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa469f6640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa40102170 0x7ffa4019eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 -- 172.21.15.23:0/1104461123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa40102170 msgr2=0x7ffa4019eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa40102170 0x7ffa4019eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 -- 172.21.15.23:0/1104461123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401005c0 msgr2=0x7ffa40197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401005c0 0x7ffa40197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 -- 172.21.15.23:0/1104461123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4019f520 con 0x7ffa401017b0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa459f4640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c890 tx=0x7ffa3000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30004220 con 0x7ffa401017b0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa300043c0 con 0x7ffa401017b0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4019f810 con 0x7ffa401017b0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa300049e0 con 0x7ffa401017b0 2024-03-28T15:50:23.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.575+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4019fd60 con 0x7ffa401017b0 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.576+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4019ffc0 con 0x7ffa401017b0 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.577+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffa300079b0 con 0x7ffa401017b0 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.578+0000 7ffa377fe640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.578+0000 7ffa461f5640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.578+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffa300ce5b0 con 0x7ffa401017b0 2024-03-28T15:50:23.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.578+0000 7ffa461f5640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffa40101610 tx=0x7ffa3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:23.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.581+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ffa30014030 con 0x7ffa401017b0 2024-03-28T15:50:23.810 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.809+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa401a0190 con 0x7ffa401017b0 2024-03-28T15:50:23.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.809+0000 7ffa377fe640 1 -- 172.21.15.23:0/1104461123 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8456 (secure 0 0 0) 0x7ffa3009b4a0 con 0x7ffa401017b0 2024-03-28T15:50:23.811 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:23.811 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T15:50:23.812 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 11 2024-03-28T15:50:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.811+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 msgr2=0x7ffa2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.811+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ffa40101610 tx=0x7ffa3c005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:23.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.811+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 msgr2=0x7ffa40197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.811+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c890 tx=0x7ffa3000cd50 comp rx=0 tx=0).stop 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 shutdown_connections 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ffa2c0a2800 0x7ffa2c0a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffa40102170 0x7ffa4019eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffa401017b0 0x7ffa40197e40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 --2- 172.21.15.23:0/1104461123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa401005c0 0x7ffa40197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 >> 172.21.15.23:0/1104461123 conn(0x7ffa400fbb30 msgr2=0x7ffa400fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 shutdown_connections 2024-03-28T15:50:23.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:23.812+0000 7ffa48459640 1 -- 172.21.15.23:0/1104461123 wait complete. 2024-03-28T15:50:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: Deploying daemon mds.a on smithi023 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6838/1641251810,v1:172.21.15.23:6839/1641251810] up:boot 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: fsmap 10 up:standby 2024-03-28T15:50:23.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:23 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: Deploying daemon mds.a on smithi023 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6838/1641251810,v1:172.21.15.23:6839/1641251810] up:boot 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: fsmap 10 up:standby 2024-03-28T15:50:23.823 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:23 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:23.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:23.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:23.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: Deploying daemon mds.a on smithi023 2024-03-28T15:50:23.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6838/1641251810,v1:172.21.15.23:6839/1641251810] up:boot 2024-03-28T15:50:23.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: fsmap 10 up:standby 2024-03-28T15:50:23.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:23 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:50:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:24 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1104461123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:24.836 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:24 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1104461123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:24.836 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:24 smithi023 systemd[1]: Starting Ceph mds.a for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:24 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1104461123' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:25.088 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:24 smithi023 podman[73533]: 2024-03-28 15:50:24.738954547 +0000 UTC m=+0.017965631 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:25.088 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:24 smithi023 podman[73533]: 2024-03-28 15:50:24.858440997 +0000 UTC m=+0.137452119 container create 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T15:50:25.088 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 podman[73533]: 2024-03-28 15:50:25.086054853 +0000 UTC m=+0.365065937 container init 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-28T15:50:25.088 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 podman[73533]: 2024-03-28 15:50:25.08850716 +0000 UTC m=+0.367518251 container start 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T15:50:25.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 ceph-mds[73552]: -- 172.21.15.23:0/2222888151 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56388da75a40 con 0x56388da49000 2024-03-28T15:50:25.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: starting mds.a at 2024-03-28T15:50:25.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 bash[73533]: 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf 2024-03-28T15:50:25.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 15:50:25 smithi023 systemd[1]: Started Ceph mds.a for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:25.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/1369060605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08104af0 msgr2=0x7f4b08106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/1369060605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08104af0 0x7f4b08106ee0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009900 tx=0x7f4afc02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:25.941 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/1369060605 shutdown_connections 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/1369060605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08107420 0x7f4b08109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/1369060605 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08104af0 0x7f4b08106ee0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/1369060605 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b081021c0 0x7f4b081045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/1369060605 >> 172.21.15.23:0/1369060605 conn(0x7f4b080fbbd0 msgr2=0x7f4b080fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/1369060605 shutdown_connections 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/1369060605 wait complete. 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 Processor -- start 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- start start 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b081021c0 0x7f4b08101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b08104af0 0x7f4b08100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.938+0000 7f4b0ff7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0810d360 con 0x7f4b08107420 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0ff7e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0810d1e0 con 0x7f4b081021c0 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0ff7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0810d4e0 con 0x7f4b08104af0 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39582/0 (socket says 172.21.15.23:39582) 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 -- 172.21.15.23:0/3843881475 learned_addr learned my addr 172.21.15.23:0/3843881475 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0dd1a640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b081021c0 0x7f4b08101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0d519640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b08104af0 0x7f4b08100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 -- 172.21.15.23:0/3843881475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b08104af0 msgr2=0x7f4b08100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b08104af0 0x7f4b08100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 -- 172.21.15.23:0/3843881475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b081021c0 msgr2=0x7f4b08101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b081021c0 0x7f4b08101b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 -- 172.21.15.23:0/3843881475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b08100f40 con 0x7f4b08107420 2024-03-28T15:50:25.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0e51b640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4af800eaa0 tx=0x7f4af800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af800cdb0 con 0x7f4b08107420 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4af8005020 con 0x7f4b08107420 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b081a1b30 con 0x7f4b08107420 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af80116b0 con 0x7f4b08107420 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.939+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b081a2050 con 0x7f4b08107420 2024-03-28T15:50:25.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.940+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b08068910 con 0x7f4b08107420 2024-03-28T15:50:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.941+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4af8005560 con 0x7f4b08107420 2024-03-28T15:50:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.942+0000 7f4b06ffd640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 0x7f4af00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.942+0000 7f4b0dd1a640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 0x7f4af00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.942+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4af80cebb0 con 0x7f4b08107420 2024-03-28T15:50:25.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.942+0000 7f4b0dd1a640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 0x7f4af00a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4b00005fd0 tx=0x7f4b0000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:25.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:25.945+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b08068910 con 0x7f4b08107420 2024-03-28T15:50:26.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.167+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b080fd710 con 0x7f4b08107420 2024-03-28T15:50:26.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.168+0000 7f4b06ffd640 1 -- 172.21.15.23:0/3843881475 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9245 (secure 0 0 0) 0x7f4b080fd710 con 0x7f4b08107420 2024-03-28T15:50:26.169 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:26.170 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-28T15:50:26.170 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 12 2024-03-28T15:50:26.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 msgr2=0x7f4af00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:26.171 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 0x7f4af00a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4b00005fd0 tx=0x7f4b0000b390 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 msgr2=0x7f4b081006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4af800eaa0 tx=0x7f4af800ef60 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 shutdown_connections 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4af00a2880 0x7f4af00a4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b08107420 0x7f4b081006b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b08104af0 0x7f4b08100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 --2- 172.21.15.23:0/3843881475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b081021c0 0x7f4b08101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 >> 172.21.15.23:0/3843881475 conn(0x7f4b080fbbd0 msgr2=0x7f4b08102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 shutdown_connections 2024-03-28T15:50:26.172 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:26.170+0000 7f4b0ff7e640 1 -- 172.21.15.23:0/3843881475 wait complete. 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: Deploying daemon mds.b on smithi117 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:boot 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: fsmap 11 up:standby 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:50:26.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:26 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3843881475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:26.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: Deploying daemon mds.b on smithi117 2024-03-28T15:50:26.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:boot 2024-03-28T15:50:26.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: fsmap 11 up:standby 2024-03-28T15:50:26.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:50:26.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:26 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3843881475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-28T15:50:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: Deploying daemon mds.b on smithi117 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:boot 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: fsmap 11 up:standby 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:50:26.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:26 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3843881475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:27.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 systemd[1]: Starting Ceph mds.b for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T15:50:27.932 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 podman[66309]: 2024-03-28 15:50:27.568760212 +0000 UTC m=+0.017558156 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T15:50:27.933 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 podman[66309]: 2024-03-28 15:50:27.688273476 +0000 UTC m=+0.137071407 container create cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T15:50:27.933 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 podman[66309]: 2024-03-28 15:50:27.932973097 +0000 UTC m=+0.381771027 container init cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-28T15:50:28.183 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:28.236 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 podman[66309]: 2024-03-28 15:50:27.935893846 +0000 UTC m=+0.384691782 container start cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True) 2024-03-28T15:50:28.236 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 ceph-mds[66327]: -- 172.21.15.117:0/4080309546 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ee63d7a40 con 0x559ee63ab000 2024-03-28T15:50:28.236 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: starting mds.b at 2024-03-28T15:50:28.237 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:28 smithi117 bash[66309]: cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb 2024-03-28T15:50:28.237 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:28 smithi117 systemd[1]: Started Ceph mds.b for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T15:50:28.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 -- 172.21.15.23:0/3433844760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a0102170 msgr2=0x7fc4a0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 --2- 172.21.15.23:0/3433844760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a0102170 0x7fc4a0108a00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc498009ca0 tx=0x7fc49802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 -- 172.21.15.23:0/3433844760 shutdown_connections 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 --2- 172.21.15.23:0/3433844760 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a0102170 0x7fc4a0108a00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 --2- 172.21.15.23:0/3433844760 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a01017b0 0x7fc4a0101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 --2- 172.21.15.23:0/3433844760 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a01009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 -- 172.21.15.23:0/3433844760 >> 172.21.15.23:0/3433844760 conn(0x7fc4a00fbb30 msgr2=0x7fc4a00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 -- 172.21.15.23:0/3433844760 shutdown_connections 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.300+0000 7fc4a7044640 1 -- 172.21.15.23:0/3433844760 wait complete. 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 Processor -- start 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 -- start start 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a0102170 0x7fc4a01a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010c240 con 0x7fc4a01017b0 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010c0c0 con 0x7fc4a01005c0 2024-03-28T15:50:28.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a7044640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a010c3c0 con 0x7fc4a0102170 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a4de0640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39600/0 (socket says 172.21.15.23:39600) 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 -- 172.21.15.23:0/4157244886 learned_addr learned my addr 172.21.15.23:0/4157244886 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a55e1640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a0102170 0x7fc4a01a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 -- 172.21.15.23:0/4157244886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a0102170 msgr2=0x7fc4a01a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a0102170 0x7fc4a01a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 -- 172.21.15.23:0/4157244886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 msgr2=0x7fc4a0069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a0069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc49ffff640 1 -- 172.21.15.23:0/4157244886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a01a17f0 con 0x7fc4a01017b0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.301+0000 7fc4a4de0640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc49ffff640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00e9f0 tx=0x7fc48c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c00cd80 con 0x7fc4a01017b0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a01a1a80 con 0x7fc4a01017b0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc48c005500 con 0x7fc4a01017b0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c010690 con 0x7fc4a01017b0 2024-03-28T15:50:28.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.302+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01a1f80 con 0x7fc4a01017b0 2024-03-28T15:50:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.303+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0107880 con 0x7fc4a01017b0 2024-03-28T15:50:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.304+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc48c0046c0 con 0x7fc4a01017b0 2024-03-28T15:50:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.305+0000 7fc49dffb640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 0x7fc4900a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.305+0000 7fc4a4de0640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 0x7fc4900a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.305+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc48c0ce990 con 0x7fc4a01017b0 2024-03-28T15:50:28.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.305+0000 7fc4a4de0640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 0x7fc4900a4cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0101610 tx=0x7fc494005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:28.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.310+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc48c09bed0 con 0x7fc4a01017b0 2024-03-28T15:50:28.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.535+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4a002d070 con 0x7fc4a01017b0 2024-03-28T15:50:28.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.536+0000 7fc49dffb640 1 -- 172.21.15.23:0/4157244886 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10040 (secure 0 0 0) 0x7fc48c09b7d0 con 0x7fc4a01017b0 2024-03-28T15:50:28.537 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:28.537 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24494,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-28T15:50:28.537 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 13 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 msgr2=0x7fc4900a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 0x7fc4900a4cf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0101610 tx=0x7fc494005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 msgr2=0x7fc4a019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00e9f0 tx=0x7fc48c00edf0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 shutdown_connections 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4900a2830 0x7fc4900a4cf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4a0102170 0x7fc4a01a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4a01017b0 0x7fc4a019eab0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 --2- 172.21.15.23:0/4157244886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4a01005c0 0x7fc4a0069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 >> 172.21.15.23:0/4157244886 conn(0x7fc4a00fbb30 msgr2=0x7fc4a0104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 shutdown_connections 2024-03-28T15:50:28.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.538+0000 7fc4a7044640 1 -- 172.21.15.23:0/4157244886 wait complete. 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.550 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:28 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.552 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:28 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:28 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:28.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 -- 172.21.15.23:0/4253131111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081017b0 msgr2=0x7fb608101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 --2- 172.21.15.23:0/4253131111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081017b0 0x7fb608101c30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009900 tx=0x7fb5fc02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:28.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 -- 172.21.15.23:0/4253131111 shutdown_connections 2024-03-28T15:50:28.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 --2- 172.21.15.23:0/4253131111 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb608102170 0x7fb608108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 --2- 172.21.15.23:0/4253131111 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081017b0 0x7fb608101c30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 --2- 172.21.15.23:0/4253131111 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6081005c0 0x7fb6081009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 -- 172.21.15.23:0/4253131111 >> 172.21.15.23:0/4253131111 conn(0x7fb6080fbb30 msgr2=0x7fb6080fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 -- 172.21.15.23:0/4253131111 shutdown_connections 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.656+0000 7fb60ccde640 1 -- 172.21.15.23:0/4253131111 wait complete. 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 Processor -- start 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 -- start start 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6081017b0 0x7fb608197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb608102170 0x7fb60819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60810c240 con 0x7fb6081005c0 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60810c0c0 con 0x7fb608102170 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60ccde640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb60810c3c0 con 0x7fb6081017b0 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58438/0 (socket says 172.21.15.23:58438) 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 -- 172.21.15.23:0/1071122827 learned_addr learned my addr 172.21.15.23:0/1071122827 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb605d9b640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6081017b0 0x7fb608197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb606d9d640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb608102170 0x7fb60819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 -- 172.21.15.23:0/1071122827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6081017b0 msgr2=0x7fb608197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6081017b0 0x7fb608197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 -- 172.21.15.23:0/1071122827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb608102170 msgr2=0x7fb60819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb608102170 0x7fb60819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 -- 172.21.15.23:0/1071122827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb60819f600 con 0x7fb6081005c0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.657+0000 7fb60659c640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e9c0 tx=0x7fb5f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.658+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f800cd80 con 0x7fb6081005c0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.658+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb60819f8f0 con 0x7fb6081005c0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.658+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5f80035f0 con 0x7fb6081005c0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.658+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5f8010690 con 0x7fb6081005c0 2024-03-28T15:50:28.661 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.658+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb60819fe10 con 0x7fb6081005c0 2024-03-28T15:50:28.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.659+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb608068490 con 0x7fb6081005c0 2024-03-28T15:50:28.662 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.660+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5f80037a0 con 0x7fb6081005c0 2024-03-28T15:50:28.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.660+0000 7fb5f77fe640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 0x7fb5f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:28.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.660+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb5f80ce970 con 0x7fb6081005c0 2024-03-28T15:50:28.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.660+0000 7fb605d9b640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 0x7fb5f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:28.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.661+0000 7fb605d9b640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 0x7fb5f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc02c7b0 tx=0x7fb5fc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:28.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.665+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5f809beb0 con 0x7fb6081005c0 2024-03-28T15:50:28.884 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.882+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6081a00a0 con 0x7fb6081005c0 2024-03-28T15:50:28.885 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.884+0000 7fb5f77fe640 1 -- 172.21.15.23:0/1071122827 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18481 (secure 0 0 0) 0x7fb5f809b7b0 con 0x7fb6081005c0 2024-03-28T15:50:28.885 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:28.885 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":82,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:49:45.294119+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 msgr2=0x7fb5f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 0x7fb5f00a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc02c7b0 tx=0x7fb5fc005fd0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 msgr2=0x7fb6081979e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb5f800e9c0 tx=0x7fb5f800ee80 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 shutdown_connections 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb5f00a2800 0x7fb5f00a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb608102170 0x7fb60819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6081017b0 0x7fb608197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 --2- 172.21.15.23:0/1071122827 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6081005c0 0x7fb6081979e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:28.890 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 >> 172.21.15.23:0/1071122827 conn(0x7fb6080fbb30 msgr2=0x7fb6080fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:28.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 shutdown_connections 2024-03-28T15:50:28.891 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:28.886+0000 7fb60ccde640 1 -- 172.21.15.23:0/1071122827 wait complete. 2024-03-28T15:50:28.903 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-28T15:50:28.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:50:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- 172.21.15.23:0/2826972546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 msgr2=0x7f3e141097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 --2- 172.21.15.23:0/2826972546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e141097f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c009ca0 tx=0x7f3e0c02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- 172.21.15.23:0/2826972546 shutdown_connections 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 --2- 172.21.15.23:0/2826972546 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e141097f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 --2- 172.21.15.23:0/2826972546 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14104a70 0x7f3e14106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 --2- 172.21.15.23:0/2826972546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14102140 0x7f3e14104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- 172.21.15.23:0/2826972546 >> 172.21.15.23:0/2826972546 conn(0x7f3e140fbb50 msgr2=0x7f3e140fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- 172.21.15.23:0/2826972546 shutdown_connections 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- 172.21.15.23:0/2826972546 wait complete. 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 Processor -- start 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.008+0000 7f3e195f4640 1 -- start start 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14102140 0x7f3e141979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14104a70 0x7f3e14197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1410d360 con 0x7f3e141073a0 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1410d1e0 con 0x7f3e14104a70 2024-03-28T15:50:29.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1410d4e0 con 0x7f3e14102140 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58456/0 (socket says 172.21.15.23:58456) 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 -- 172.21.15.23:0/788466313 learned_addr learned my addr 172.21.15.23:0/788466313 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e12ffd640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14102140 0x7f3e141979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e0bfff640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14104a70 0x7f3e14197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 -- 172.21.15.23:0/788466313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14102140 msgr2=0x7f3e141979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14102140 0x7f3e141979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 -- 172.21.15.23:0/788466313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14104a70 msgr2=0x7f3e14197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14104a70 0x7f3e14197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 -- 172.21.15.23:0/788466313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1419f5c0 con 0x7f3e141073a0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e137fe640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02c700 tx=0x7f3e0c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c039040 con 0x7f3e141073a0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e0c044070 con 0x7f3e141073a0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0c0042a0 con 0x7f3e141073a0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1419f850 con 0x7f3e141073a0 2024-03-28T15:50:29.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.009+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1419fd40 con 0x7f3e141073a0 2024-03-28T15:50:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.010+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e140fd640 con 0x7f3e141073a0 2024-03-28T15:50:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.011+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3e0c003970 con 0x7f3e141073a0 2024-03-28T15:50:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.012+0000 7f3e10ff9640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 0x7f3e000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.012+0000 7f3e12ffd640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 0x7f3e000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.012+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3e0c0f0b30 con 0x7f3e141073a0 2024-03-28T15:50:29.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.012+0000 7f3e12ffd640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 0x7f3e000a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bbe0 tx=0x7f3e0400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.016+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3e0c0be100 con 0x7f3e141073a0 2024-03-28T15:50:29.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.200+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e140622a0 con 0x7f3e141073a0 2024-03-28T15:50:29.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.201+0000 7f3e10ff9640 1 -- 172.21.15.23:0/788466313 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e0c0bda00 con 0x7f3e141073a0 2024-03-28T15:50:29.203 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.204+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 msgr2=0x7f3e000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.204+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 0x7f3e000a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400bbe0 tx=0x7f3e0400b390 comp rx=0 tx=0).stop 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.204+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 msgr2=0x7f3e1419ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.204+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c02c700 tx=0x7f3e0c032f20 comp rx=0 tx=0).stop 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 shutdown_connections 2024-03-28T15:50:29.206 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3e000a2800 0x7f3e000a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3e141073a0 0x7f3e1419ef50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e14104a70 0x7f3e14197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 --2- 172.21.15.23:0/788466313 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3e14102140 0x7f3e141979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 >> 172.21.15.23:0/788466313 conn(0x7f3e140fbb50 msgr2=0x7f3e14102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 shutdown_connections 2024-03-28T15:50:29.207 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.205+0000 7f3e195f4640 1 -- 172.21.15.23:0/788466313 wait complete. 2024-03-28T15:50:29.222 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-28T15:50:29.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 -- 172.21.15.23:0/2970630128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 msgr2=0x7fef78109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 --2- 172.21.15.23:0/2970630128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef78109800 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fef70009ca0 tx=0x7fef7002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 -- 172.21.15.23:0/2970630128 shutdown_connections 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 --2- 172.21.15.23:0/2970630128 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef78109800 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 --2- 172.21.15.23:0/2970630128 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78104a80 0x7fef78106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 --2- 172.21.15.23:0/2970630128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78102150 0x7fef78104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 -- 172.21.15.23:0/2970630128 >> 172.21.15.23:0/2970630128 conn(0x7fef780fbb80 msgr2=0x7fef780fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 -- 172.21.15.23:0/2970630128 shutdown_connections 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 -- 172.21.15.23:0/2970630128 wait complete. 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.327+0000 7fef7e787640 1 Processor -- start 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 -- start start 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78102150 0x7fef78197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78104a80 0x7fef78197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7810d350 con 0x7fef781073b0 2024-03-28T15:50:29.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7810d1d0 con 0x7fef78104a80 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7e787640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7810d4d0 con 0x7fef78102150 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef77fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78102150 0x7fef78197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58482/0 (socket says 172.21.15.23:58482) 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 -- 172.21.15.23:0/3159625372 learned_addr learned my addr 172.21.15.23:0/3159625372 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef777fe640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78104a80 0x7fef78197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 -- 172.21.15.23:0/3159625372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78102150 msgr2=0x7fef78197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78102150 0x7fef78197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 -- 172.21.15.23:0/3159625372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78104a80 msgr2=0x7fef78197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78104a80 0x7fef78197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 -- 172.21.15.23:0/3159625372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7819f590 con 0x7fef781073b0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.328+0000 7fef7cd24640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef7002c700 tx=0x7fef70032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.329+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef70039040 con 0x7fef781073b0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.329+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef70044070 con 0x7fef781073b0 2024-03-28T15:50:29.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.329+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7819f820 con 0x7fef781073b0 2024-03-28T15:50:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.329+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef700042a0 con 0x7fef781073b0 2024-03-28T15:50:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.329+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7819fd10 con 0x7fef781073b0 2024-03-28T15:50:29.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.330+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef780fd670 con 0x7fef781073b0 2024-03-28T15:50:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.331+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fef70003970 con 0x7fef781073b0 2024-03-28T15:50:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.331+0000 7fef757fa640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 0x7fef600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.331+0000 7fef77fff640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 0x7fef600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.331+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fef700f0a80 con 0x7fef781073b0 2024-03-28T15:50:29.334 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.332+0000 7fef77fff640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 0x7fef600a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef68005d00 tx=0x7fef6800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.336+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef700be050 con 0x7fef781073b0 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.523+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fef780622a0 con 0x7fef781073b0 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.524+0000 7fef757fa640 1 -- 172.21.15.23:0/3159625372 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7fef700bd950 con 0x7fef781073b0 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stdout:[ 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stdout: "poolnum": 1, 2024-03-28T15:50:29.525 INFO:teuthology.orchestra.run.smithi023.stdout: "poolname": ".mgr" 2024-03-28T15:50:29.526 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T15:50:29.526 INFO:teuthology.orchestra.run.smithi023.stdout:] 2024-03-28T15:50:29.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 msgr2=0x7fef600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 0x7fef600a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fef68005d00 tx=0x7fef6800b410 comp rx=0 tx=0).stop 2024-03-28T15:50:29.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 msgr2=0x7fef7819ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fef7002c700 tx=0x7fef70032f20 comp rx=0 tx=0).stop 2024-03-28T15:50:29.527 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 shutdown_connections 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef600a2800 0x7fef600a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef781073b0 0x7fef7819ef20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef78104a80 0x7fef78197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 --2- 172.21.15.23:0/3159625372 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef78102150 0x7fef78197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 >> 172.21.15.23:0/3159625372 conn(0x7fef780fbb80 msgr2=0x7fef78102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 shutdown_connections 2024-03-28T15:50:29.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.526+0000 7fef7e787640 1 -- 172.21.15.23:0/3159625372 wait complete. 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:boot 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: fsmap 12 up:standby 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4157244886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.537 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1071122827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.538 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:29 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/788466313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:29.538 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-28T15:50:29.539 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:boot 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: fsmap 12 up:standby 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4157244886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1071122827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:29 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/788466313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:boot 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: fsmap 12 up:standby 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4157244886' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1071122827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:29.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:29 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/788466313' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:29.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 -- 172.21.15.23:0/3535454902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24104a60 msgr2=0x7f5b24106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 --2- 172.21.15.23:0/3535454902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24104a60 0x7f5b24106e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c009900 tx=0x7f5b0c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:29.648 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 -- 172.21.15.23:0/3535454902 shutdown_connections 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 --2- 172.21.15.23:0/3535454902 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24107390 0x7f5b241097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 --2- 172.21.15.23:0/3535454902 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24104a60 0x7f5b24106e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 --2- 172.21.15.23:0/3535454902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 -- 172.21.15.23:0/3535454902 >> 172.21.15.23:0/3535454902 conn(0x7f5b240fbb40 msgr2=0x7f5b240fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 -- 172.21.15.23:0/3535454902 shutdown_connections 2024-03-28T15:50:29.649 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.647+0000 7f5b289d4640 1 -- 172.21.15.23:0/3535454902 wait complete. 2024-03-28T15:50:29.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 Processor -- start 2024-03-28T15:50:29.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 -- start start 2024-03-28T15:50:29.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 0x7f5b2419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2410d330 con 0x7f5b24107390 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2410d1b0 con 0x7f5b24102130 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b289d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2410d4b0 con 0x7f5b24104a60 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b22d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b22d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58502/0 (socket says 172.21.15.23:58502) 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b22d9d640 1 -- 172.21.15.23:0/2190564489 learned_addr learned my addr 172.21.15.23:0/2190564489 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 0x7f5b2419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b2259c640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 -- 172.21.15.23:0/2190564489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 msgr2=0x7f5b24199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 -- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 msgr2=0x7f5b241a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b1bfff640 1 -- 172.21.15.23:0/2190564489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b241a1850 con 0x7f5b24104a60 2024-03-28T15:50:29.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b2259c640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.648+0000 7f5b22d9d640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b1bfff640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 0x7f5b2419a180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c02c170 tx=0x7f5b0c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c041070 con 0x7f5b24104a60 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241a1ae0 con 0x7f5b24104a60 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b0c004450 con 0x7f5b24104a60 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c003e50 con 0x7f5b24104a60 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.649+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b241a1ff0 con 0x7f5b24104a60 2024-03-28T15:50:29.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.650+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24108ee0 con 0x7f5b24104a60 2024-03-28T15:50:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.651+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5b0c044070 con 0x7f5b24104a60 2024-03-28T15:50:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.651+0000 7f5b1b7fe640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 0x7f5b100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.652+0000 7f5b2259c640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 0x7f5b100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.652+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5b0c0f00a0 con 0x7f5b24104a60 2024-03-28T15:50:29.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.652+0000 7f5b2259c640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 0x7f5b100a4e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5b2419b110 tx=0x7f5b14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:29.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.656+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b0c0bcf90 con 0x7f5b24104a60 2024-03-28T15:50:29.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:29.859+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f5b2402d270 con 0x7f5b24104a60 2024-03-28T15:50:30.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.391+0000 7f5b1b7fe640 1 -- 172.21.15.23:0/2190564489 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5b0c0c1030 con 0x7f5b24104a60 2024-03-28T15:50:30.394 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' created 2024-03-28T15:50:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 msgr2=0x7f5b100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 0x7f5b100a4e60 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5b2419b110 tx=0x7f5b14005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 msgr2=0x7f5b2419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 0x7f5b2419a180 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c02c170 tx=0x7f5b0c02c650 comp rx=0 tx=0).stop 2024-03-28T15:50:30.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 shutdown_connections 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5b100a29a0 0x7f5b100a4e60 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5b24107390 0x7f5b241a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5b24104a60 0x7f5b2419a180 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 --2- 172.21.15.23:0/2190564489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b24102130 0x7f5b24199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 >> 172.21.15.23:0/2190564489 conn(0x7f5b240fbb40 msgr2=0x7f5b24101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 shutdown_connections 2024-03-28T15:50:30.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.395+0000 7f5b289d4640 1 -- 172.21.15.23:0/2190564489 wait complete. 2024-03-28T15:50:30.407 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 -- 172.21.15.23:0/2865421259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178103320 msgr2=0x7f0178109bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 --2- 172.21.15.23:0/2865421259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178103320 0x7f0178109bb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0170009cd0 tx=0x7f017002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 -- 172.21.15.23:0/2865421259 shutdown_connections 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 --2- 172.21.15.23:0/2865421259 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178103320 0x7f0178109bb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 --2- 172.21.15.23:0/2865421259 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178102960 0x7f0178102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 --2- 172.21.15.23:0/2865421259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 0x7f01780ffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 -- 172.21.15.23:0/2865421259 >> 172.21.15.23:0/2865421259 conn(0x7f01780fb2a0 msgr2=0x7f01780fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 -- 172.21.15.23:0/2865421259 shutdown_connections 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.516+0000 7f017d697640 1 -- 172.21.15.23:0/2865421259 wait complete. 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 Processor -- start 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 -- start start 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 0x7f01781022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178103320 0x7f0178100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017810d340 con 0x7f0178102960 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017810d1c0 con 0x7f01780ff990 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f017d697640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017810d4c0 con 0x7f0178103320 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58512/0 (socket says 172.21.15.23:58512) 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 -- 172.21.15.23:0/2881827503 learned_addr learned my addr 172.21.15.23:0/2881827503 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:30.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f01777fe640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178103320 0x7f0178100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f0176ffd640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 0x7f01781022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 -- 172.21.15.23:0/2881827503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178103320 msgr2=0x7f0178100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178103320 0x7f0178100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 -- 172.21.15.23:0/2881827503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 msgr2=0x7f01781022d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 0x7f01781022d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 -- 172.21.15.23:0/2881827503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01781016e0 con 0x7f0178102960 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.517+0000 7f016ffff640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f016000d950 tx=0x7f016000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.518+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0160017030 con 0x7f0178102960 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.518+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01781a1aa0 con 0x7f0178102960 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.518+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0160004020 con 0x7f0178102960 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.518+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01600047a0 con 0x7f0178102960 2024-03-28T15:50:30.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.518+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01781a2040 con 0x7f0178102960 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.519+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0178108a90 con 0x7f0178102960 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.520+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01600041d0 con 0x7f0178102960 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.521+0000 7f0174ff9640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 0x7f01640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.521+0000 7f0176ffd640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 0x7f01640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.521+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f01600ce760 con 0x7f0178102960 2024-03-28T15:50:30.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.521+0000 7f0176ffd640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 0x7f01640a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0178069ab0 tx=0x7f0168005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:30.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.525+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f016009bb30 con 0x7f0178102960 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3159625372' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2190564489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:30 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3159625372' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2190564489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:30 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3159625372' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2190564489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:30 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:30.710 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:30.709+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f01780622a0 con 0x7f0178102960 2024-03-28T15:50:31.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.401+0000 7f0174ff9640 1 -- 172.21.15.23:0/2881827503 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v84) v1 ==== 171+0+0 (secure 0 0 0) 0x7f016009b430 con 0x7f0178102960 2024-03-28T15:50:31.402 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' created 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 msgr2=0x7f01640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 0x7f01640a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0178069ab0 tx=0x7f0168005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 msgr2=0x7f0178100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f016000d950 tx=0x7f016000de10 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 shutdown_connections 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01640a2800 0x7f01640a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0178103320 0x7f0178100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0178102960 0x7f0178100910 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 --2- 172.21.15.23:0/2881827503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01780ff990 0x7f01781022d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 >> 172.21.15.23:0/2881827503 conn(0x7f01780fb2a0 msgr2=0x7f0178105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 shutdown_connections 2024-03-28T15:50:31.406 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.404+0000 7f017d697640 1 -- 172.21.15.23:0/2881827503 wait complete. 2024-03-28T15:50:31.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 -- 172.21.15.23:0/1977217976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8101790 msgr2=0x7f01b8101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 --2- 172.21.15.23:0/1977217976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8101790 0x7f01b8101c10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009900 tx=0x7f01ac02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 -- 172.21.15.23:0/1977217976 shutdown_connections 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 --2- 172.21.15.23:0/1977217976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8102150 0x7f01b81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 --2- 172.21.15.23:0/1977217976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8101790 0x7f01b8101c10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 --2- 172.21.15.23:0/1977217976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 -- 172.21.15.23:0/1977217976 >> 172.21.15.23:0/1977217976 conn(0x7f01b80fbb30 msgr2=0x7f01b80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 -- 172.21.15.23:0/1977217976 shutdown_connections 2024-03-28T15:50:31.540 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.536+0000 7f01bd587640 1 -- 172.21.15.23:0/1977217976 wait complete. 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 Processor -- start 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 -- start start 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8101790 0x7f01b819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 0x7f01b819ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b810c250 con 0x7f01b8102150 2024-03-28T15:50:31.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b810c0d0 con 0x7f01b8101790 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01bd587640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b810c3d0 con 0x7f01b81005a0 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 0x7f01b819ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b6ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:37954/0 (socket says 172.21.15.23:37954) 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b6ffd640 1 -- 172.21.15.23:0/317041177 learned_addr learned my addr 172.21.15.23:0/317041177 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b67fc640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8101790 0x7f01b819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b77fe640 1 -- 172.21.15.23:0/317041177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 msgr2=0x7f01b8069390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.537+0000 7f01b77fe640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01b77fe640 1 -- 172.21.15.23:0/317041177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8101790 msgr2=0x7f01b819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:31.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01b77fe640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8101790 0x7f01b819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01b77fe640 1 -- 172.21.15.23:0/317041177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b819f620 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01b6ffd640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01b77fe640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 0x7f01b819ed60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01b000da10 tx=0x7f01b000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b000bd70 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b819f910 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01b0003e50 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b00106a0 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.538+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b819fe30 con 0x7f01b8102150 2024-03-28T15:50:31.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.539+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b8068490 con 0x7f01b8102150 2024-03-28T15:50:31.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.541+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01b0004b40 con 0x7f01b8102150 2024-03-28T15:50:31.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.542+0000 7f019ffff640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 0x7f01a00a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:31.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.542+0000 7f01b6ffd640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 0x7f01a00a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:31.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.542+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f01b00cefc0 con 0x7f01b8102150 2024-03-28T15:50:31.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.542+0000 7f01b6ffd640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 0x7f01a00a90d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01a80058c0 tx=0x7f01a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:31.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.545+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f01b009c210 con 0x7f01b8102150 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:31.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:31.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:31 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:31.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:31.782+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f01b802d070 con 0x7f01b8102150 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:31.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:31 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: osdmap e83: 12 total, 12 up, 12 in 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:31.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:31.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:31 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:50:32.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.412+0000 7f019ffff640 1 -- 172.21.15.23:0/317041177 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f01b009bb10 con 0x7f01b8102150 2024-03-28T15:50:32.421 INFO:teuthology.orchestra.run.smithi023.stderr:new fs with metadata pool 2 and data pool 3 2024-03-28T15:50:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.429+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 msgr2=0x7f01a00a90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.429+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 0x7f01a00a90d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f01a80058c0 tx=0x7f01a8005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:32.432 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.429+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 msgr2=0x7f01b819ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.429+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 0x7f01b819ed60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f01b000da10 tx=0x7f01b000ded0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 shutdown_connections 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f01a00a6c10 0x7f01a00a90d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f01b8102150 0x7f01b819ed60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01b8101790 0x7f01b819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 --2- 172.21.15.23:0/317041177 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f01b81005a0 0x7f01b8069390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 >> 172.21.15.23:0/317041177 conn(0x7f01b80fbb30 msgr2=0x7f01b8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 shutdown_connections 2024-03-28T15:50:32.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.431+0000 7f01bd587640 1 -- 172.21.15.23:0/317041177 wait complete. 2024-03-28T15:50:32.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:32 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T15:50:32.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:32 smithi023 ceph-mon[32448]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T15:50:32.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:32 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T15:50:32.449 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:32 smithi023 ceph-mon[32448]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T15:50:32.450 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-28T15:50:32.450 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 -- 172.21.15.23:0/1568144761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97801073c0 msgr2=0x7f9780109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 --2- 172.21.15.23:0/1568144761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97801073c0 0x7f9780109810 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9778009ca0 tx=0x7f977802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 -- 172.21.15.23:0/1568144761 shutdown_connections 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 --2- 172.21.15.23:0/1568144761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f97801073c0 0x7f9780109810 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 --2- 172.21.15.23:0/1568144761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9780104a90 0x7f9780106e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 --2- 172.21.15.23:0/1568144761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 0x7f9780104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 -- 172.21.15.23:0/1568144761 >> 172.21.15.23:0/1568144761 conn(0x7f97800fbb50 msgr2=0x7f97800fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 -- 172.21.15.23:0/1568144761 shutdown_connections 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.567+0000 7f9785cff640 1 -- 172.21.15.23:0/1568144761 wait complete. 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 Processor -- start 2024-03-28T15:50:32.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 -- start start 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 0x7f9780197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 0x7f978019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978010d380 con 0x7f9780104a90 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978010d200 con 0x7f9780102160 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f9785cff640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f978010d500 con 0x7f97801073c0 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f977effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:32.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f977effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58564/0 (socket says 172.21.15.23:58564) 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f977effd640 1 -- 172.21.15.23:0/2350680957 learned_addr learned my addr 172.21.15.23:0/2350680957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f977f7fe640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 0x7f9780197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.568+0000 7f977ffff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 0x7f978019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 -- 172.21.15.23:0/2350680957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 msgr2=0x7f978019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 0x7f978019f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 -- 172.21.15.23:0/2350680957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 msgr2=0x7f9780197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 0x7f9780197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 -- 172.21.15.23:0/2350680957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978019f670 con 0x7f9780104a90 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977ffff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 0x7f978019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977effd640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f977400e9f0 tx=0x7f977400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f977400ccc0 con 0x7f9780104a90 2024-03-28T15:50:32.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f978019f960 con 0x7f9780104a90 2024-03-28T15:50:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9774005500 con 0x7f9780104a90 2024-03-28T15:50:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9774010690 con 0x7f9780104a90 2024-03-28T15:50:32.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.569+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978019feb0 con 0x7f9780104a90 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.570+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97800fd640 con 0x7f9780104a90 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.571+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f97740046c0 con 0x7f9780104a90 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.572+0000 7f977cff9640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 0x7f97680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.572+0000 7f977f7fe640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 0x7f97680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.572+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f97740cee60 con 0x7f9780104a90 2024-03-28T15:50:32.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.573+0000 7f977f7fe640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 0x7f97680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9770009bd0 tx=0x7f9770005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:32.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.576+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9774053740 con 0x7f9780104a90 2024-03-28T15:50:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:32 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T15:50:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:32 smithi120 ceph-mon[36826]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T15:50:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:32 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T15:50:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:32 smithi120 ceph-mon[36826]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T15:50:32.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:32.817+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f978002d070 con 0x7f9780104a90 2024-03-28T15:50:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:32 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2881827503' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-28T15:50:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:32 smithi117 ceph-mon[36002]: osdmap e84: 12 total, 12 up, 12 in 2024-03-28T15:50:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:32 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-28T15:50:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:32 smithi117 ceph-mon[36002]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-28T15:50:33.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.418+0000 7f977cff9640 1 -- 172.21.15.23:0/2350680957 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v86) v1 ==== 157+0+0 (secure 0 0 0) 0x7f977409c050 con 0x7f9780104a90 2024-03-28T15:50:33.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 msgr2=0x7f97680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 0x7f97680a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9770009bd0 tx=0x7f9770005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 msgr2=0x7f9780197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f977400e9f0 tx=0x7f977400eeb0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 shutdown_connections 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f97680a2800 0x7f97680a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f97801073c0 0x7f978019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9780104a90 0x7f9780197f90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 --2- 172.21.15.23:0/2350680957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9780102160 0x7f9780197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.420+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 >> 172.21.15.23:0/2350680957 conn(0x7f97800fbb50 msgr2=0x7f9780102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:33.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.421+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 shutdown_connections 2024-03-28T15:50:33.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.421+0000 7f9785cff640 1 -- 172.21.15.23:0/2350680957 wait complete. 2024-03-28T15:50:33.435 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-28T15:50:33.547 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 -- 172.21.15.23:0/2620990582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 msgr2=0x7f69a8104650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 --2- 172.21.15.23:0/2620990582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8104650 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009900 tx=0x7f69a402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 -- 172.21.15.23:0/2620990582 shutdown_connections 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 --2- 172.21.15.23:0/2620990582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a81074c0 0x7f69a81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 --2- 172.21.15.23:0/2620990582 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a8104b90 0x7f69a8106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 --2- 172.21.15.23:0/2620990582 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8104650 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 -- 172.21.15.23:0/2620990582 >> 172.21.15.23:0/2620990582 conn(0x7f69a80fbbb0 msgr2=0x7f69a80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 -- 172.21.15.23:0/2620990582 shutdown_connections 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.544+0000 7f69af91b640 1 -- 172.21.15.23:0/2620990582 wait complete. 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 Processor -- start 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 -- start start 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a8104b90 0x7f69a8197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:33.548 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a81074c0 0x7f69a819efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a810d370 con 0x7f69a8102260 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a810d1f0 con 0x7f69a8104b90 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69af91b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a810d4f0 con 0x7f69a81074c0 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58586/0 (socket says 172.21.15.23:58586) 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 -- 172.21.15.23:0/1869597365 learned_addr learned my addr 172.21.15.23:0/1869597365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69adeb8640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a81074c0 0x7f69a819efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69aceb6640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a8104b90 0x7f69a8197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 -- 172.21.15.23:0/1869597365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a81074c0 msgr2=0x7f69a819efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a81074c0 0x7f69a819efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.549 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 -- 172.21.15.23:0/1869597365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a8104b90 msgr2=0x7f69a8197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a8104b90 0x7f69a8197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 -- 172.21.15.23:0/1869597365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a819f630 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.545+0000 7f69ad6b7640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009230 tx=0x7f69a4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.546+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4041070 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.546+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69a40044d0 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.546+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4034440 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.546+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a819f8c0 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.546+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a819fe00 con 0x7f69a8102260 2024-03-28T15:50:33.550 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.547+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a80fd6a0 con 0x7f69a8102260 2024-03-28T15:50:33.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.548+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f69a4044070 con 0x7f69a8102260 2024-03-28T15:50:33.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.548+0000 7f699e7fc640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 0x7f69940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:33.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.548+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f69a40f05b0 con 0x7f69a8102260 2024-03-28T15:50:33.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.548+0000 7f69aceb6640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 0x7f69940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:33.551 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.549+0000 7f69aceb6640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 0x7f69940a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f69a8198f20 tx=0x7f699800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:33.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.553+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f69a40bcfd0 con 0x7f69a8102260 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: pgmap v283: 97 pgs: 21 creating+peering, 67 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:33.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:33.738+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f69a802d070 con 0x7f69a8102260 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: pgmap v283: 97 pgs: 21 creating+peering, 67 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:33.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: pgmap v283: 97 pgs: 21 creating+peering, 67 unknown, 9 active+clean; 577 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: osdmap e85: 12 total, 12 up, 12 in 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/317041177' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:50:33.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:active 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:34 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:active 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:34 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2350680957' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: osdmap e86: 12 total, 12 up, 12 in 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] up:active 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:34 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-28T15:50:35.431 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.431+0000 7f699e7fc640 1 -- 172.21.15.23:0/1869597365 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v88) v1 ==== 228+0+0 (secure 0 0 0) 0x7f69a40c1070 con 0x7f69a8102260 2024-03-28T15:50:35.432 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' created 2024-03-28T15:50:35.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 msgr2=0x7f69940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 0x7f69940a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f69a8198f20 tx=0x7f699800b390 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 msgr2=0x7f69a8197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009230 tx=0x7f69a4005d50 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 shutdown_connections 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f69940a2880 0x7f69940a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f69a81074c0 0x7f69a819efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69a8104b90 0x7f69a8197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 --2- 172.21.15.23:0/1869597365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f69a8102260 0x7f69a8197a10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 >> 172.21.15.23:0/1869597365 conn(0x7f69a80fbbb0 msgr2=0x7f69a8102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 shutdown_connections 2024-03-28T15:50:35.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.434+0000 7f69af91b640 1 -- 172.21.15.23:0/1869597365 wait complete. 2024-03-28T15:50:35.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-28T15:50:35.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 -- 172.21.15.23:0/881884483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 msgr2=0x7f5db41097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 --2- 172.21.15.23:0/881884483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db41097f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5dac009ca0 tx=0x7f5dac02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 -- 172.21.15.23:0/881884483 shutdown_connections 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 --2- 172.21.15.23:0/881884483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db41097f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 --2- 172.21.15.23:0/881884483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db4104ad0 0x7f5db4106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 --2- 172.21.15.23:0/881884483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db41021a0 0x7f5db4104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 -- 172.21.15.23:0/881884483 >> 172.21.15.23:0/881884483 conn(0x7f5db40fbb50 msgr2=0x7f5db40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 -- 172.21.15.23:0/881884483 shutdown_connections 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.563+0000 7f5db9c36640 1 -- 172.21.15.23:0/881884483 wait complete. 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 Processor -- start 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 -- start start 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 0x7f5db41979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:35.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 0x7f5db4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db410d360 con 0x7f5db4107400 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db410d1e0 con 0x7f5db41021a0 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db9c36640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db410d4e0 con 0x7f5db4104ad0 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58606/0 (socket says 172.21.15.23:58606) 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 -- 172.21.15.23:0/2968645695 learned_addr learned my addr 172.21.15.23:0/2968645695 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db37fe640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 0x7f5db41979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db2ffd640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 0x7f5db4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 -- 172.21.15.23:0/2968645695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 msgr2=0x7f5db4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 0x7f5db4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 -- 172.21.15.23:0/2968645695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 msgr2=0x7f5db41979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 0x7f5db41979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db3fff640 1 -- 172.21.15.23:0/2968645695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db419f5f0 con 0x7f5db4107400 2024-03-28T15:50:35.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db37fe640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 0x7f5db41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.564+0000 7f5db2ffd640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 0x7f5db4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db3fff640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5dac02c630 tx=0x7f5dac032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac041020 con 0x7f5db4107400 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5db419f880 con 0x7f5db4107400 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5dac034cd0 con 0x7f5db4107400 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac039040 con 0x7f5db4107400 2024-03-28T15:50:35.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.565+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5db419fdc0 con 0x7f5db4107400 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.566+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5db40fd640 con 0x7f5db4107400 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.567+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5dac044020 con 0x7f5db4107400 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.568+0000 7f5db0ff9640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 0x7f5d9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.568+0000 7f5db37fe640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 0x7f5d9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.568+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f5dac0f1030 con 0x7f5db4107400 2024-03-28T15:50:35.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.568+0000 7f5db37fe640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 0x7f5d9c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005fd0 tx=0x7f5da800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:35.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.571+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5dac04b020 con 0x7f5db4107400 2024-03-28T15:50:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:35.756+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f5db40622a0 con 0x7f5db4107400 2024-03-28T15:50:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:35 smithi117 ceph-mon[36002]: pgmap v286: 97 pgs: 21 creating+peering, 21 unknown, 55 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-28T15:50:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:35 smithi117 ceph-mon[36002]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T15:50:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:35 smithi117 ceph-mon[36002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T15:50:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:35 smithi117 ceph-mon[36002]: Cluster is now healthy 2024-03-28T15:50:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:35 smithi120 ceph-mon[36826]: pgmap v286: 97 pgs: 21 creating+peering, 21 unknown, 55 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-28T15:50:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:35 smithi120 ceph-mon[36826]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T15:50:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:35 smithi120 ceph-mon[36826]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T15:50:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:35 smithi120 ceph-mon[36826]: Cluster is now healthy 2024-03-28T15:50:35.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:35 smithi023 ceph-mon[32448]: pgmap v286: 97 pgs: 21 creating+peering, 21 unknown, 55 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 8 op/s 2024-03-28T15:50:35.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:35 smithi023 ceph-mon[32448]: osdmap e87: 12 total, 12 up, 12 in 2024-03-28T15:50:35.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:35 smithi023 ceph-mon[32448]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-28T15:50:35.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:35 smithi023 ceph-mon[32448]: Cluster is now healthy 2024-03-28T15:50:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.437+0000 7f5db0ff9640 1 -- 172.21.15.23:0/2968645695 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v89) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5dac0bdeb0 con 0x7f5db4107400 2024-03-28T15:50:36.437 INFO:teuthology.orchestra.run.smithi023.stderr:set pool 4 allow_ec_overwrites to true 2024-03-28T15:50:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 msgr2=0x7f5d9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 0x7f5d9c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5da8005fd0 tx=0x7f5da800b390 comp rx=0 tx=0).stop 2024-03-28T15:50:36.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 msgr2=0x7f5db419ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5dac02c630 tx=0x7f5dac032ea0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 shutdown_connections 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5d9c0a2800 0x7f5d9c0a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5db4107400 0x7f5db419ef80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5db4104ad0 0x7f5db4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 --2- 172.21.15.23:0/2968645695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5db41021a0 0x7f5db41979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 >> 172.21.15.23:0/2968645695 conn(0x7f5db40fbb50 msgr2=0x7f5db4102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 shutdown_connections 2024-03-28T15:50:36.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.441+0000 7f5db9c36640 1 -- 172.21.15.23:0/2968645695 wait complete. 2024-03-28T15:50:36.463 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-28T15:50:36.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 -- 172.21.15.23:0/2418267669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2741016b0 msgr2=0x7fb274101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 --2- 172.21.15.23:0/2418267669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2741016b0 0x7fb274101ad0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb264009900 tx=0x7fb26402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 -- 172.21.15.23:0/2418267669 shutdown_connections 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 --2- 172.21.15.23:0/2418267669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb274103260 0x7fb274109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 --2- 172.21.15.23:0/2418267669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 0x7fb274102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 --2- 172.21.15.23:0/2418267669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb2741016b0 0x7fb274101ad0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 -- 172.21.15.23:0/2418267669 >> 172.21.15.23:0/2418267669 conn(0x7fb2740fcc40 msgr2=0x7fb2740ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 -- 172.21.15.23:0/2418267669 shutdown_connections 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 -- 172.21.15.23:0/2418267669 wait complete. 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.573+0000 7fb279997640 1 Processor -- start 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 -- start start 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2741016b0 0x7fb274197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 0x7fb274197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:36.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27410d370 con 0x7fb274103260 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27410d1f0 con 0x7fb2741016b0 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb279997640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb27410d4f0 con 0x7fb2741028a0 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58628/0 (socket says 172.21.15.23:58628) 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 -- 172.21.15.23:0/3450846015 learned_addr learned my addr 172.21.15.23:0/3450846015 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb272ffd640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2741016b0 0x7fb274197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2727fc640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 0x7fb274197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 -- 172.21.15.23:0/3450846015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 msgr2=0x7fb274197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 0x7fb274197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 -- 172.21.15.23:0/3450846015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2741016b0 msgr2=0x7fb274197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2741016b0 0x7fb274197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:36.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 -- 172.21.15.23:0/3450846015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27419f630 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb2737fe640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00eb10 tx=0x7fb26c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.574+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c004b20 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.575+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb26c004cc0 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.575+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb27419f920 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.575+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb26c010760 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.575+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb27419fe70 con 0x7fb274103260 2024-03-28T15:50:36.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.575+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb274108a50 con 0x7fb274103260 2024-03-28T15:50:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.577+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb26c005000 con 0x7fb274103260 2024-03-28T15:50:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.577+0000 7fb278995640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 0x7fb25c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.578+0000 7fb272ffd640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 0x7fb25c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.578+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fb26c0cf450 con 0x7fb274103260 2024-03-28T15:50:36.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.578+0000 7fb272ffd640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 0x7fb25c0a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb26402c170 tx=0x7fb26402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:36.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.581+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb26c053740 con 0x7fb274103260 2024-03-28T15:50:36.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:36.807+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fb27402d2c0 con 0x7fb274103260 2024-03-28T15:50:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:36 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T15:50:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:36 smithi117 ceph-mon[36002]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T15:50:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:36 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T15:50:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:36 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T15:50:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:36 smithi120 ceph-mon[36826]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T15:50:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:36 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T15:50:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:36 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1869597365' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-28T15:50:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:36 smithi023 ceph-mon[32448]: osdmap e88: 12 total, 12 up, 12 in 2024-03-28T15:50:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:36 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-28T15:50:37.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.445+0000 7fb278995640 1 -- 172.21.15.23:0/3450846015 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fb26c09c2d0 con 0x7fb274103260 2024-03-28T15:50:37.446 INFO:teuthology.orchestra.run.smithi023.stderr:added data pool 4 to fsmap 2024-03-28T15:50:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 msgr2=0x7fb25c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 0x7fb25c0a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb26402c170 tx=0x7fb26402df90 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 msgr2=0x7fb27419efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00eb10 tx=0x7fb26c00c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 shutdown_connections 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb25c0a2880 0x7fb25c0a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb274103260 0x7fb27419efc0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb2741028a0 0x7fb274197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 --2- 172.21.15.23:0/3450846015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2741016b0 0x7fb274197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.448+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 >> 172.21.15.23:0/3450846015 conn(0x7fb2740fcc40 msgr2=0x7fb274105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.449+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 shutdown_connections 2024-03-28T15:50:37.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.449+0000 7fb279997640 1 -- 172.21.15.23:0/3450846015 wait complete. 2024-03-28T15:50:37.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:37.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 -- 172.21.15.23:0/3895312369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701017d0 msgr2=0x7f7070101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 --2- 172.21.15.23:0/3895312369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701017d0 0x7f7070101c50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7060009900 tx=0x7f706002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 -- 172.21.15.23:0/3895312369 shutdown_connections 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 --2- 172.21.15.23:0/3895312369 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7070102190 0x7f7070108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 --2- 172.21.15.23:0/3895312369 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f70701017d0 0x7f7070101c50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 --2- 172.21.15.23:0/3895312369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f7070100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 -- 172.21.15.23:0/3895312369 >> 172.21.15.23:0/3895312369 conn(0x7f70700fbb50 msgr2=0x7f70700fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 -- 172.21.15.23:0/3895312369 shutdown_connections 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 -- 172.21.15.23:0/3895312369 wait complete. 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.570+0000 7f70775b2640 1 Processor -- start 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 -- start start 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f707006a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 0x7f707019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010c2c0 con 0x7f7070102190 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010c140 con 0x7f70701005e0 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707010c440 con 0x7f70701017d0 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58644/0 (socket says 172.21.15.23:58644) 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 -- 172.21.15.23:0/776803244 learned_addr learned my addr 172.21.15.23:0/776803244 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f707534e640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f707006a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7074b4d640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 0x7f707019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 -- 172.21.15.23:0/776803244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 msgr2=0x7f707019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 0x7f707019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 -- 172.21.15.23:0/776803244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 msgr2=0x7f707006a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.574 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f707006a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 -- 172.21.15.23:0/776803244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707019f610 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7074b4d640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 0x7f707019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f707534e640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f707006a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f7075b4f640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f706800cad0 tx=0x7f706800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800ae20 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7068005020 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707019f900 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.571+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706800f6f0 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.572+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707019fe50 con 0x7f7070102190 2024-03-28T15:50:37.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.573+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7070107900 con 0x7f7070102190 2024-03-28T15:50:37.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.574+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f70680051d0 con 0x7f7070102190 2024-03-28T15:50:37.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.574+0000 7f70667fc640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 0x7f705c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.574+0000 7f707534e640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 0x7f705c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.574+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f70680cf020 con 0x7f7070102190 2024-03-28T15:50:37.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.575+0000 7f707534e640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 0x7f705c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7070101630 tx=0x7f706c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:37.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.579+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7068014030 con 0x7f7070102190 2024-03-28T15:50:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.801+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70700622a0 con 0x7f7070102190 2024-03-28T15:50:37.801 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.801+0000 7f70667fc640 1 -- 172.21.15.23:0/776803244 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11324 (secure 0 0 0) 0x7f706809be80 con 0x7f7070102190 2024-03-28T15:50:37.802 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:37.802 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:50:36.808900+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-03-28T15:50:37.802 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 msgr2=0x7f705c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 0x7f705c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7070101630 tx=0x7f706c005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 msgr2=0x7f707019ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f706800cad0 tx=0x7f706800cf90 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 shutdown_connections 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f705c0a2800 0x7f705c0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7070102190 0x7f707019ed50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f70701017d0 0x7f707019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 --2- 172.21.15.23:0/776803244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70701005e0 0x7f707006a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 >> 172.21.15.23:0/776803244 conn(0x7f70700fbb50 msgr2=0x7f7070104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:37.804 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 shutdown_connections 2024-03-28T15:50:37.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.803+0000 7f70775b2640 1 -- 172.21.15.23:0/776803244 wait complete. 2024-03-28T15:50:37.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:37 smithi023 ceph-mon[32448]: pgmap v289: 129 pgs: 32 unknown, 21 creating+peering, 76 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-28T15:50:37.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:37 smithi023 ceph-mon[32448]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:37.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:37 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T15:50:37.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:37 smithi023 ceph-mon[32448]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T15:50:37.813 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:37 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T15:50:37.815 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:37.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:37 smithi117 ceph-mon[36002]: pgmap v289: 129 pgs: 32 unknown, 21 creating+peering, 76 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-28T15:50:37.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:37 smithi117 ceph-mon[36002]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:37.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:37 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T15:50:37.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:37 smithi117 ceph-mon[36002]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T15:50:37.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:37 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T15:50:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:37 smithi120 ceph-mon[36826]: pgmap v289: 129 pgs: 32 unknown, 21 creating+peering, 76 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-28T15:50:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:37 smithi120 ceph-mon[36826]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-28T15:50:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:37 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2968645695' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-28T15:50:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:37 smithi120 ceph-mon[36826]: osdmap e89: 12 total, 12 up, 12 in 2024-03-28T15:50:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:37 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-28T15:50:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2569494000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4481028c0 msgr2=0x7fc448102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2569494000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4481028c0 0x7fc448102d40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc434009900 tx=0x7fc43402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2569494000 shutdown_connections 2024-03-28T15:50:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2569494000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc448103280 0x7fc448109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.920 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2569494000 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc4481028c0 0x7fc448102d40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2569494000 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2569494000 >> 172.21.15.23:0/2569494000 conn(0x7fc4480fcc40 msgr2=0x7fc4480ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2569494000 shutdown_connections 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2569494000 wait complete. 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.917+0000 7fc44ea6c640 1 Processor -- start 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 -- start start 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4481028c0 0x7fc448197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44810d350 con 0x7fc448103280 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44810d1d0 con 0x7fc4481028c0 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44810d4d0 con 0x7fc4481016d0 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.921 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58658/0 (socket says 172.21.15.23:58658) 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 -- 172.21.15.23:0/2656424528 learned_addr learned my addr 172.21.15.23:0/2656424528 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc447fff640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4481028c0 0x7fc448197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44c808640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 -- 172.21.15.23:0/2656424528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 msgr2=0x7fc448197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 -- 172.21.15.23:0/2656424528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4481028c0 msgr2=0x7fc448197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4481028c0 0x7fc448197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 -- 172.21.15.23:0/2656424528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44819f640 con 0x7fc448103280 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44c808640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44d009640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc44000eb10 tx=0x7fc44000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440004b20 con 0x7fc448103280 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44819f930 con 0x7fc448103280 2024-03-28T15:50:37.922 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc440004cc0 con 0x7fc448103280 2024-03-28T15:50:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.918+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440010760 con 0x7fc448103280 2024-03-28T15:50:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.919+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44819fe00 con 0x7fc448103280 2024-03-28T15:50:37.923 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.920+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc448108a40 con 0x7fc448103280 2024-03-28T15:50:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.920+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc440005020 con 0x7fc448103280 2024-03-28T15:50:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.921+0000 7fc445ffb640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 0x7fc4380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.921+0000 7fc44c808640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 0x7fc4380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.921+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc4400cf220 con 0x7fc448103280 2024-03-28T15:50:37.924 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.922+0000 7fc44c808640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 0x7fc4380a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0058c0 tx=0x7fc43c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:37.926 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:37.926+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc44009c120 con 0x7fc448103280 2024-03-28T15:50:38.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.110+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc44802d290 con 0x7fc448103280 2024-03-28T15:50:38.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.111+0000 7fc445ffb640 1 -- 172.21.15.23:0/2656424528 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7fc44009ba20 con 0x7fc448103280 2024-03-28T15:50:38.111 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:38.112 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:38.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 msgr2=0x7fc4380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 0x7fc4380a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0058c0 tx=0x7fc43c005850 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 msgr2=0x7fc44819efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc44000eb10 tx=0x7fc44000c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 shutdown_connections 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc4380a2800 0x7fc4380a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc448103280 0x7fc44819efd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4481028c0 0x7fc448197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 --2- 172.21.15.23:0/2656424528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc4481016d0 0x7fc448197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 >> 172.21.15.23:0/2656424528 conn(0x7fc4480fcc40 msgr2=0x7fc448105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 shutdown_connections 2024-03-28T15:50:38.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.113+0000 7fc44ea6c640 1 -- 172.21.15.23:0/2656424528 wait complete. 2024-03-28T15:50:38.125 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/2744441999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 msgr2=0x7f6d58101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/2744441999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58101c50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009900 tx=0x7f6d4802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:38.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/2744441999 shutdown_connections 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/2744441999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 0x7f6d58108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/2744441999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58101c50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/2744441999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 0x7f6d58100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/2744441999 >> 172.21.15.23:0/2744441999 conn(0x7f6d580fbb50 msgr2=0x7f6d580fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/2744441999 shutdown_connections 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/2744441999 wait complete. 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 Processor -- start 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.228+0000 7f6d5e5d4640 1 -- start start 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 0x7f6d58104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 0x7f6d581030f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5810c260 con 0x7f6d581017d0 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5810c0e0 con 0x7f6d58102190 2024-03-28T15:50:38.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5810c3e0 con 0x7f6d581005e0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58680/0 (socket says 172.21.15.23:58680) 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 -- 172.21.15.23:0/3994755265 learned_addr learned my addr 172.21.15.23:0/3994755265 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d57fff640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 0x7f6d58104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5cb71640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 0x7f6d581030f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 -- 172.21.15.23:0/3994755265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 msgr2=0x7f6d58104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 0x7f6d58104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 -- 172.21.15.23:0/3994755265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 msgr2=0x7f6d581030f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 0x7f6d581030f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 -- 172.21.15.23:0/3994755265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d58103980 con 0x7f6d581017d0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d577fe640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009230 tx=0x7f6d48005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48041070 con 0x7f6d581017d0 2024-03-28T15:50:38.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d581a1ac0 con 0x7f6d581017d0 2024-03-28T15:50:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.229+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d480044d0 con 0x7f6d581017d0 2024-03-28T15:50:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.230+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d48034440 con 0x7f6d581017d0 2024-03-28T15:50:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.230+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d581a2010 con 0x7f6d581017d0 2024-03-28T15:50:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.230+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d581a21e0 con 0x7f6d581017d0 2024-03-28T15:50:38.233 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.232+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6d48044070 con 0x7f6d581017d0 2024-03-28T15:50:38.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.232+0000 7f6d557fa640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.232+0000 7f6d57fff640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.232+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6d480f0b70 con 0x7f6d581017d0 2024-03-28T15:50:38.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.233+0000 7f6d57fff640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 0x7f6d440a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005fd0 tx=0x7f6d4000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.236 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.236+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d480bd380 con 0x7f6d581017d0 2024-03-28T15:50:38.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.420+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d581a23b0 con 0x7f6d581017d0 2024-03-28T15:50:38.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.422+0000 7f6d557fa640 1 -- 172.21.15.23:0/3994755265 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7f6d480c1420 con 0x7f6d581017d0 2024-03-28T15:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:38.422 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 msgr2=0x7f6d440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 0x7f6d440a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6d40005fd0 tx=0x7f6d4000b390 comp rx=0 tx=0).stop 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 msgr2=0x7f6d58102bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6d48009230 tx=0x7f6d48005d50 comp rx=0 tx=0).stop 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 shutdown_connections 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6d440a2800 0x7f6d440a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d58102190 0x7f6d581030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6d581017d0 0x7f6d58102bb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 --2- 172.21.15.23:0/3994755265 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6d581005e0 0x7f6d58104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 >> 172.21.15.23:0/3994755265 conn(0x7f6d580fbb50 msgr2=0x7f6d580fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 shutdown_connections 2024-03-28T15:50:38.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.424+0000 7f6d5e5d4640 1 -- 172.21.15.23:0/3994755265 wait complete. 2024-03-28T15:50:38.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 -- 172.21.15.23:0/852614322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c0ff990 msgr2=0x7fef9c0ffdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 --2- 172.21.15.23:0/852614322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c0ff990 0x7fef9c0ffdb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009930 tx=0x7fef8c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 -- 172.21.15.23:0/852614322 shutdown_connections 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 --2- 172.21.15.23:0/852614322 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c103320 0x7fef9c109bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 --2- 172.21.15.23:0/852614322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c102960 0x7fef9c102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.554 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 --2- 172.21.15.23:0/852614322 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c0ff990 0x7fef9c0ffdb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 -- 172.21.15.23:0/852614322 >> 172.21.15.23:0/852614322 conn(0x7fef9c0fb2c0 msgr2=0x7fef9c0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 -- 172.21.15.23:0/852614322 shutdown_connections 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.552+0000 7fefa1187640 1 -- 172.21.15.23:0/852614322 wait complete. 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 Processor -- start 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 -- start start 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c0ff990 0x7fef9c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c103320 0x7fef9c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c10d3b0 con 0x7fef9c102960 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c10d230 con 0x7fef9c103320 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fefa1187640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c10d530 con 0x7fef9c0ff990 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43490/0 (socket says 172.21.15.23:43490) 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 -- 172.21.15.23:0/2086708067 learned_addr learned my addr 172.21.15.23:0/2086708067 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9b59e640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c103320 0x7fef9c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9ad9d640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c0ff990 0x7fef9c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 -- 172.21.15.23:0/2086708067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c0ff990 msgr2=0x7fef9c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c0ff990 0x7fef9c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 -- 172.21.15.23:0/2086708067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c103320 msgr2=0x7fef9c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c103320 0x7fef9c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.553+0000 7fef9a59c640 1 -- 172.21.15.23:0/2086708067 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c19f640 con 0x7fef9c102960 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fef9a59c640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000cd80 con 0x7fef9c102960 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c19f930 con 0x7fef9c102960 2024-03-28T15:50:38.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef90010910 con 0x7fef9c102960 2024-03-28T15:50:38.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef900188f0 con 0x7fef9c102960 2024-03-28T15:50:38.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.554+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c19fed0 con 0x7fef9c102960 2024-03-28T15:50:38.557 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.555+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c108b10 con 0x7fef9c102960 2024-03-28T15:50:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.556+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fef90010ac0 con 0x7fef9c102960 2024-03-28T15:50:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.556+0000 7fef83fff640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 0x7fef840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.557+0000 7fef9ad9d640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 0x7fef840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.557+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fef900cf040 con 0x7fef9c102960 2024-03-28T15:50:38.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.557+0000 7fef9ad9d640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 0x7fef840a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009230 tx=0x7fef8c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.561 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.561+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef9009b850 con 0x7fef9c102960 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T15:50:38.254+0000 7f6e41e4c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/776803244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2656424528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:38 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3994755265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.785+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef9c02d070 con 0x7fef9c102960 2024-03-28T15:50:38.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.785+0000 7fef83fff640 1 -- 172.21.15.23:0/2086708067 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11324 (secure 0 0 0) 0x7fef9009f8f0 con 0x7fef9c102960 2024-03-28T15:50:38.786 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:38.786 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24392,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24403,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24422,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24442,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24454,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24455,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24481,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:50:36.808900+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24494},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24494,"qdb_cluster":[24494]},"id":1}]} 2024-03-28T15:50:38.786 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 17 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 msgr2=0x7fef840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 0x7fef840a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fef8c009230 tx=0x7fef8c02df90 comp rx=0 tx=0).stop 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 msgr2=0x7fef9c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fef9000e9f0 tx=0x7fef9000eeb0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 shutdown_connections 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fef840a2800 0x7fef840a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef9c103320 0x7fef9c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fef9c102960 0x7fef9c197f60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 --2- 172.21.15.23:0/2086708067 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fef9c0ff990 0x7fef9c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 >> 172.21.15.23:0/2086708067 conn(0x7fef9c0fb2c0 msgr2=0x7fef9c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 shutdown_connections 2024-03-28T15:50:38.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.788+0000 7fefa1187640 1 -- 172.21.15.23:0/2086708067 wait complete. 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/776803244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2656424528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.798 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:38 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3994755265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.800 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: osdmap e90: 12 total, 12 up, 12 in 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3450846015' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/776803244' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2656424528' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:38 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3994755265' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:38.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 -- 172.21.15.23:0/1093885019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f80104af0 msgr2=0x7f9f80106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1093885019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f80104af0 0x7f9f80106ee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009860 tx=0x7f9f7402bb70 comp rx=0 tx=0).stop 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 -- 172.21.15.23:0/1093885019 shutdown_connections 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1093885019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80107420 0x7f9f80109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1093885019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f80104af0 0x7f9f80106ee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.901+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1093885019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f801021c0 0x7f9f801045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- 172.21.15.23:0/1093885019 >> 172.21.15.23:0/1093885019 conn(0x7f9f800fbb30 msgr2=0x7f9f800fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- 172.21.15.23:0/1093885019 shutdown_connections 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- 172.21.15.23:0/1093885019 wait complete. 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 Processor -- start 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- start start 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 0x7f9f80197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80104af0 0x7f9f80197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8010d340 con 0x7f9f80107420 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8010d1c0 con 0x7f9f801021c0 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f878ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8010d4c0 con 0x7f9f80104af0 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f85e49640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f85648640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 0x7f9f80197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f85e49640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43514/0 (socket says 172.21.15.23:43514) 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.902+0000 7f9f85e49640 1 -- 172.21.15.23:0/1488666071 learned_addr learned my addr 172.21.15.23:0/1488666071 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f84e47640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80104af0 0x7f9f80197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 -- 172.21.15.23:0/1488666071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80104af0 msgr2=0x7f9f80197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80104af0 0x7f9f80197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 -- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 msgr2=0x7f9f8019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:38.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 -- 172.21.15.23:0/1488666071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8019f630 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85e49640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f85648640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 0x7f9f80197a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e790 tx=0x7f9f7800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7800eea0 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f7800ccc0 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78018780 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8019f920 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.903+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8019fe40 con 0x7f9f801021c0 2024-03-28T15:50:38.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.904+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f80068490 con 0x7f9f801021c0 2024-03-28T15:50:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.905+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9f78010430 con 0x7f9f801021c0 2024-03-28T15:50:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.905+0000 7f9f7e7fc640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 0x7f9f680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.906+0000 7f9f84e47640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 0x7f9f680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.906+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9f780ceea0 con 0x7f9f801021c0 2024-03-28T15:50:38.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.906+0000 7f9f84e47640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 0x7f9f680a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009830 tx=0x7f9f74035bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:38.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:38.910+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f7809bd00 con 0x7f9f801021c0 2024-03-28T15:50:39.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.094+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f800622d0 con 0x7f9f801021c0 2024-03-28T15:50:39.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.095+0000 7f9f7e7fc640 1 -- 172.21.15.23:0/1488666071 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7f9f7809b600 con 0x7f9f801021c0 2024-03-28T15:50:39.096 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:39.096 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 msgr2=0x7f9f680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 0x7f9f680a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009830 tx=0x7f9f74035bc0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 msgr2=0x7f9f80197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 0x7f9f80197a10 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800e790 tx=0x7f9f7800ec50 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 shutdown_connections 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9f680a2800 0x7f9f680a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f80107420 0x7f9f8019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f80104af0 0x7f9f80197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 --2- 172.21.15.23:0/1488666071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f801021c0 0x7f9f80197a10 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 >> 172.21.15.23:0/1488666071 conn(0x7f9f800fbb30 msgr2=0x7f9f80102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 shutdown_connections 2024-03-28T15:50:39.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.097+0000 7f9f878ac640 1 -- 172.21.15.23:0/1488666071 wait complete. 2024-03-28T15:50:39.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-28T15:50:31.785133+0000', 'modified': '2024-03-28T15:50:36.808900+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24494}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3971000510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3971000510}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3971000510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24494, 'qdb_cluster': [24494]} 2024-03-28T15:50:39.109 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-28T15:50:39.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- 172.21.15.23:0/3205409422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c08104af0 msgr2=0x7f7c08106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/3205409422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c08104af0 0x7f7c08106ee0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009900 tx=0x7f7bf402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- 172.21.15.23:0/3205409422 shutdown_connections 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/3205409422 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08107420 0x7f7c08109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/3205409422 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c08104af0 0x7f7c08106ee0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.215 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/3205409422 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c081021c0 0x7f7c081045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- 172.21.15.23:0/3205409422 >> 172.21.15.23:0/3205409422 conn(0x7f7c080fbb30 msgr2=0x7f7c080fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- 172.21.15.23:0/3205409422 shutdown_connections 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- 172.21.15.23:0/3205409422 wait complete. 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 Processor -- start 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- start start 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08104af0 0x7f7c08100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c08107420 0x7f7c081006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0810d2f0 con 0x7f7c081021c0 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0810d170 con 0x7f7c08104af0 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0ec30640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0810d470 con 0x7f7c08107420 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0c9cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.216 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0c9cc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43534/0 (socket says 172.21.15.23:43534) 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.213+0000 7f7c0c9cc640 1 -- 172.21.15.23:0/2265346020 learned_addr learned my addr 172.21.15.23:0/2265346020 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c07fff640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08104af0 0x7f7c08100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c08107420 0x7f7c081006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 -- 172.21.15.23:0/2265346020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c08107420 msgr2=0x7f7c081006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c08107420 0x7f7c081006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 -- 172.21.15.23:0/2265346020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08104af0 msgr2=0x7f7c08100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08104af0 0x7f7c08100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 -- 172.21.15.23:0/2265346020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08100f40 con 0x7f7c081021c0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00d950 tx=0x7f7bfc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c05ffb640 1 -- 172.21.15.23:0/2265346020 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc017030 con 0x7f7c081021c0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c05ffb640 1 -- 172.21.15.23:0/2265346020 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bfc004020 con 0x7f7c081021c0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c081a1b20 con 0x7f7c081021c0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c05ffb640 1 -- 172.21.15.23:0/2265346020 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bfc0047a0 con 0x7f7c081021c0 2024-03-28T15:50:39.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.214+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c081a2040 con 0x7f7c081021c0 2024-03-28T15:50:39.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.216+0000 7f7c05ffb640 1 -- 172.21.15.23:0/2265346020 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7bfc0041d0 con 0x7f7c081021c0 2024-03-28T15:50:39.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 Processor -- start 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 -- start start 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffba740 0x555b0ffbab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffbb130 0x555b0ffbfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ff1fc20 con 0x555b0ffc02a0 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0fec8a40 con 0x555b0ffba740 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c11817740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ff1bcc0 con 0x555b0ffbb130 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c05ffb640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 0x7f7bf80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43544/0 (socket says 172.21.15.23:43544) 2024-03-28T15:50:39.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 -- 172.21.15.23:0/32936463 learned_addr learned my addr 172.21.15.23:0/32936463 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffba740 0x555b0ffbab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffbb130 0x555b0ffbfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 0x7f7bf80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c05ffb640 1 -- 172.21.15.23:0/2265346020 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7bfc0ce690 con 0x7f7c081021c0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 -- 172.21.15.23:0/32936463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffbb130 msgr2=0x555b0ffbfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffbb130 0x555b0ffbfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.217+0000 7f7c07fff640 1 -- 172.21.15.23:0/32936463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffba740 msgr2=0x555b0ffbab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c07fff640 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffba740 0x555b0ffbab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c07fff640 1 -- 172.21.15.23:0/32936463 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b0feeaf90 con 0x555b0ffc02a0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c07fff640 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c080692c0 tx=0x7f7bf40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2d8adb4b388ec07 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 0x7f7bf80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x555b0ffbb990 tx=0x7f7c00009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7bf2ffd640 1 -- 172.21.15.23:0/32936463 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf402cb80 con 0x555b0ffc02a0 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7bf2ffd640 1 -- 172.21.15.23:0/32936463 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bf4005040 con 0x555b0ffc02a0 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7bf2ffd640 1 -- 172.21.15.23:0/32936463 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4004900 con 0x555b0ffc02a0 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c11817740 1 -- 172.21.15.23:0/32936463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 msgr2=0x555b0ffc2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.218+0000 7f7c11817740 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7c080692c0 tx=0x7f7bf40095e0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- 172.21.15.23:0/32936463 shutdown_connections 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffc2730 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffbb130 0x555b0ffbfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- 172.21.15.23:0/32936463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffba740 0x555b0ffbab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- 172.21.15.23:0/32936463 >> 172.21.15.23:0/32936463 conn(0x555b0ff82a60 msgr2=0x555b0ffb60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- 172.21.15.23:0/32936463 shutdown_connections 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- 172.21.15.23:0/32936463 wait complete. 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- start start 2024-03-28T15:50:39.227 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 Processor -- start 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffc31f0 con 0x555b0ffc02a0 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0fedbe90 con 0x555b0ffbb130 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffc3370 con 0x555b0ffba740 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c11817740 10 client.0 _ll_register_callbacks cb 0x555b0fc33ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c0d1cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c0d1cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43558/0 (socket says 172.21.15.23:43558) 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/485050304 learned_addr learned my addr 172.21.15.23:0/485050304 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.219+0000 7f7c07fff640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/485050304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 msgr2=0x555b0ffc4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 msgr2=0x555b0ffc5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/485050304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b0fee89d0 con 0x555b0ffc02a0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c07fff640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fef0 tx=0x7f7c00004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca19c1e399b7dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00004b20 con 0x555b0ffc02a0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7c00004cc0 con 0x555b0ffc02a0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bf4005040 con 0x555b0ffc02a0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c00005020 con 0x555b0ffc02a0 2024-03-28T15:50:39.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c11817740 10 client.14754 fetch_fsmap learned FSMap version 17 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555b0fc342d0 con 0x555b0ffc02a0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.220+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bf4005040 con 0x555b0ffc02a0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f7c0001d070 con 0x555b0ffc02a0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 10 client.14754 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 10 client.14754 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 20 client.14754 populate_metadata read hostname 'smithi023' 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b0ffd14b0 0x555b0ffd04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 4 client.14754 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b0ff2d160 con 0x555b0ffd14b0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b0ffd14b0 0x555b0ffd04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7be7fff640 10 client.14754 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.221+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b0ffd14b0 0x555b0ffd04f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0da8627d1d3b01a server_cookie=48154254b717ba76 in_seq=0 out_seq=0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7be7fff640 1 -- 172.21.15.23:0/485050304 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b0ff2d160 con 0x555b0ffd14b0 2024-03-28T15:50:39.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7be7fff640 10 client.14754 handle_command_reply: tid=0 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 client.14754 shutdown 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 msgr2=0x555b0ffcc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000fef0 tx=0x7f7c00004ab0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 shutdown_connections 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b0ffd14b0 0x555b0ffd04f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcc1b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 --2- 172.21.15.23:0/485050304 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7be7fff640 0 client.14754 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.222+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 >> 172.21.15.23:0/485050304 conn(0x555b0ff82a60 msgr2=0x555b0ffb52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.223+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 shutdown_connections 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.223+0000 7f7c11817740 1 -- 172.21.15.23:0/485050304 wait complete. 2024-03-28T15:50:39.231 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.223+0000 7f7c11817740 20 client.14754 trim_cache size 0 max 16384 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 Processor -- start 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 -- start start 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffc02a0 0x555b0ffb5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd0cd0 con 0x555b0ffbb130 2024-03-28T15:50:39.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd1330 con 0x555b0ffc02a0 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c11817740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd1990 con 0x555b0ffba740 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43572/0 (socket says 172.21.15.23:43572) 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.239+0000 7f7c07fff640 1 -- 172.21.15.23:0/3653056275 learned_addr learned my addr 172.21.15.23:0/3653056275 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffc02a0 0x555b0ffb5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 -- 172.21.15.23:0/3653056275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 msgr2=0x555b0ffc3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 -- 172.21.15.23:0/3653056275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffc02a0 msgr2=0x555b0ffb5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffc02a0 0x555b0ffb5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 -- 172.21.15.23:0/3653056275 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b0fcfe6b0 con 0x555b0ffbb130 2024-03-28T15:50:39.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7c07fff640 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7bf402c710 tx=0x7f7bf4005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c30aca0b668560bb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7bf27fc640 1 -- 172.21.15.23:0/3653056275 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4004040 con 0x555b0ffbb130 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7bf27fc640 1 -- 172.21.15.23:0/3653056275 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bf4005550 con 0x555b0ffbb130 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.240+0000 7f7bf27fc640 1 -- 172.21.15.23:0/3653056275 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4004b60 con 0x555b0ffbb130 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- 172.21.15.23:0/3653056275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 msgr2=0x555b0ffcc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7bf402c710 tx=0x7f7bf4005e50 comp rx=0 tx=0).stop 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- 172.21.15.23:0/3653056275 shutdown_connections 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffc02a0 0x555b0ffb5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffbb130 0x555b0ffcc0d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- 172.21.15.23:0/3653056275 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- 172.21.15.23:0/3653056275 >> 172.21.15.23:0/3653056275 conn(0x555b0ff82a60 msgr2=0x555b0ffbbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- 172.21.15.23:0/3653056275 shutdown_connections 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- 172.21.15.23:0/3653056275 wait complete. 2024-03-28T15:50:39.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- start start 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 Processor -- start 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd2650 con 0x555b0ffc02a0 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd1ff0 con 0x555b0ffbb130 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b0ffd2cb0 con 0x555b0ffba740 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.241+0000 7f7c11817740 10 client.0 _ll_register_callbacks cb 0x7f7c0000fd80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43574/0 (socket says 172.21.15.23:43574) 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c07fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:48930/0 (socket says 172.21.15.23:48930) 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0c9cc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.248 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/1241842017 learned_addr learned my addr 172.21.15.23:0/1241842017 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/1241842017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 msgr2=0x555b0ffc79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 msgr2=0x555b0ffc7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 -- 172.21.15.23:0/1241842017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b0ffcb4f0 con 0x555b0ffc02a0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c07fff640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c0d1cd640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c0001e530 tx=0x7f7c00013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1cf881478cea22 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7be7fff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0003e060 con 0x555b0ffc02a0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7c000051c0 con 0x555b0ffc02a0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7bf4005550 con 0x555b0ffc02a0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c00045050 con 0x555b0ffc02a0 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c11817740 10 client.14760 fetch_fsmap learned FSMap version 17 2024-03-28T15:50:39.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x555b1002c570 con 0x555b0ffc02a0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.242+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7bf4005550 con 0x555b0ffc02a0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 17) v1 ==== 7922+0+0 (secure 0 0 0) 0x7f7c0001d040 con 0x555b0ffc02a0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 10 client.14760 fetch_fsmap finished waiting for FSMap version 17 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 10 client.14760 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 20 client.14760 populate_metadata read hostname 'smithi023' 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b10030ba0 0x555b1002c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 4 client.14760 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555b0ff274b0 con 0x555b10030ba0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b10030ba0 0x555b1002c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7be7fff640 10 client.14760 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c0c9cc640 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b10030ba0 0x555b1002c7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96a4cc4ce030db53 server_cookie=74e43acc1411231d in_seq=0 out_seq=0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7be7fff640 1 -- 172.21.15.23:0/1241842017 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555b0ff274b0 con 0x555b10030ba0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7be7fff640 10 client.14760 handle_command_reply: tid=0 2024-03-28T15:50:39.250 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 client.14760 shutdown 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 msgr2=0x555b0ffcaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7c0001e530 tx=0x7f7c00013070 comp rx=0 tx=0).stop 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 shutdown_connections 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.243+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x555b10030ba0 0x555b1002c7a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x555b0ffc02a0 0x555b0ffcaf00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555b0ffbb130 0x555b0ffc7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 --2- 172.21.15.23:0/1241842017 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x555b0ffba740 0x555b0ffc79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 >> 172.21.15.23:0/1241842017 conn(0x555b0ff82a60 msgr2=0x555b0ffbc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7be7fff640 0 client.14760 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 shutdown_connections 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 1 -- 172.21.15.23:0/1241842017 wait complete. 2024-03-28T15:50:39.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c11817740 20 client.14760 trim_cache size 0 max 16384 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24494, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 6.8254229190000002, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 17, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 90, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 11.283406491999999 2024-03-28T15:50:39.252 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:50:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 msgr2=0x7f7bf80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 0x7f7bf80a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x555b0ffbb990 tx=0x7f7c00009500 comp rx=0 tx=0).stop 2024-03-28T15:50:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 msgr2=0x7f7c08101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:39.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc00d950 tx=0x7f7bfc00de10 comp rx=0 tx=0).stop 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 shutdown_connections 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7bf80a2800 0x7f7bf80a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7c08107420 0x7f7c081006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c08104af0 0x7f7c08100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 --2- 172.21.15.23:0/2265346020 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7c081021c0 0x7f7c08101b30 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 >> 172.21.15.23:0/2265346020 conn(0x7f7c080fbb30 msgr2=0x7f7c08102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 shutdown_connections 2024-03-28T15:50:39.254 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:39.244+0000 7f7c0ec30640 1 -- 172.21.15.23:0/2265346020 wait complete. 2024-03-28T15:50:39.255 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T15:50:39.255 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:39.256 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T15:50:39.498 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:50:39.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:39 smithi023 ceph-mon[32448]: pgmap v292: 129 pgs: 10 unknown, 119 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T15:50:39.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:39 smithi023 ceph-mon[32448]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T15:50:39.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:39 smithi023 ceph-mon[32448]: Cluster is now healthy 2024-03-28T15:50:39.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:39 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2086708067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:39.741 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:39 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1488666071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:39.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:39 smithi117 ceph-mon[36002]: pgmap v292: 129 pgs: 10 unknown, 119 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T15:50:39.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:39 smithi117 ceph-mon[36002]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T15:50:39.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:39 smithi117 ceph-mon[36002]: Cluster is now healthy 2024-03-28T15:50:39.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:39 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2086708067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:39.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:39 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1488666071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:39 smithi120 ceph-mon[36826]: pgmap v292: 129 pgs: 10 unknown, 119 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-03-28T15:50:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:39 smithi120 ceph-mon[36826]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-28T15:50:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:39 smithi120 ceph-mon[36826]: Cluster is now healthy 2024-03-28T15:50:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:39 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2086708067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:50:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:39 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1488666071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 -- 172.21.15.23:0/3287343558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 msgr2=0x7fc7601074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 --2- 172.21.15.23:0/3287343558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc7601074b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc750009990 tx=0x7fc750009ca0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.189 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 -- 172.21.15.23:0/3287343558 shutdown_connections 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 --2- 172.21.15.23:0/3287343558 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101a30 0x7fc76010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 --2- 172.21.15.23:0/3287343558 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc7601074b0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 --2- 172.21.15.23:0/3287343558 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101090 0x7fc7601014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 -- 172.21.15.23:0/3287343558 >> 172.21.15.23:0/3287343558 conn(0x7fc760077fc0 msgr2=0x7fc7600783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 -- 172.21.15.23:0/3287343558 shutdown_connections 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.187+0000 7fc767537700 1 -- 172.21.15.23:0/3287343558 wait complete. 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 Processor -- start 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 -- start start 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101090 0x7fc76019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 0x7fc76019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:41.190 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760111330 con 0x7fc7601070d0 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601111b0 con 0x7fc760101a30 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc767537700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7601114b0 con 0x7fc760101090 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc765ad4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc764ad2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 0x7fc76019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc765ad4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43590/0 (socket says 172.21.15.23:43590) 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc765ad4700 1 -- 172.21.15.23:0/2092244073 learned_addr learned my addr 172.21.15.23:0/2092244073 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.188+0000 7fc7652d3700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101090 0x7fc76019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 -- 172.21.15.23:0/2092244073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101090 msgr2=0x7fc76019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101090 0x7fc76019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.191 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 -- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 msgr2=0x7fc760198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 -- 172.21.15.23:0/2092244073 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc760199030 con 0x7fc760101a30 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc765ad4700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc764ad2700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 0x7fc76019eb70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc75400e9f0 tx=0x7fc75400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc75400cbd0 con 0x7fc760101a30 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc75400cd70 con 0x7fc760101a30 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc760199320 con 0x7fc760101a30 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc754018710 con 0x7fc760101a30 2024-03-28T15:50:41.192 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.189+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7601a55b0 con 0x7fc760101a30 2024-03-28T15:50:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.190+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc760102780 con 0x7fc760101a30 2024-03-28T15:50:41.193 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.191+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc754010430 con 0x7fc760101a30 2024-03-28T15:50:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.192+0000 7fc75e7fc700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 0x7fc7480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.192+0000 7fc7652d3700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 0x7fc7480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.192+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc7540ceef0 con 0x7fc760101a30 2024-03-28T15:50:41.194 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.192+0000 7fc7652d3700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 0x7fc7480a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc750006010 tx=0x7fc75001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:41.195 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.195+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc75409bcf0 con 0x7fc760101a30 2024-03-28T15:50:41.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.448+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc7601a5780 con 0x7fc760101a30 2024-03-28T15:50:41.449 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.448+0000 7fc75e7fc700 1 -- 172.21.15.23:0/2092244073 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fc75409b5f0 con 0x7fc760101a30 2024-03-28T15:50:41.449 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 msgr2=0x7fc7480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 0x7fc7480a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc750006010 tx=0x7fc75001a040 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 msgr2=0x7fc76019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 0x7fc76019eb70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc75400e9f0 tx=0x7fc75400ed00 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 shutdown_connections 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc7480a2890 0x7fc7480a4d50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc7601070d0 0x7fc760198750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc760101090 0x7fc76019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 --2- 172.21.15.23:0/2092244073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc760101a30 0x7fc76019eb70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 >> 172.21.15.23:0/2092244073 conn(0x7fc760077fc0 msgr2=0x7fc760069f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 shutdown_connections 2024-03-28T15:50:41.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:41.450+0000 7fc767537700 1 -- 172.21.15.23:0/2092244073 wait complete. 2024-03-28T15:50:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:41 smithi117 ceph-mon[36002]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s wr, 1 op/s 2024-03-28T15:50:41.821 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:41 smithi023 ceph-mon[32448]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s wr, 1 op/s 2024-03-28T15:50:41.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:41 smithi120 ceph-mon[36826]: pgmap v293: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s wr, 1 op/s 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-28T15:50:42.175 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T15:50:42.176 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T15:50:42.176 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T15:50:42.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.176 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-28T15:50:42.192 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42193sec preferred_lft 42193sec 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:50:42.193 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-28T15:50:42.194 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T15:50:42.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-28T15:50:42.194 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T15:50:42.195 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:50:42.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.366 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T15:50:42.393 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-28T15:50:42.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.395 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-28T15:50:42.488 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-28T15:50:42.489 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T15:50:42.489 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T15:50:42.489 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T15:50:42.489 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T15:50:42.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.489 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:50:42.489 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:50:42.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T15:50:42.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T15:50:42.489 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T15:50:42.490 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:50:42.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.674 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T15:50:42.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.688 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-28T15:50:42.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.743 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:50:42.743 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:50:42.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:50:42.743 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-28T15:50:42.743 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:50:42.807 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:42 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2092244073' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:50:42.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:42 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2092244073' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:50:42.825 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-28T15:50:42.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.825 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:50:42.825 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-28T15:50:42.826 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:50:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:42 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2092244073' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:50:42.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:42.959 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:50:42.959 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:50:42.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-28T15:50:42.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T15:50:42.960 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:50:43.001 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-28T15:50:43.001 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:50:43.001 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:43.056 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-28T15:50:43.056 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:50:43.056 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:43.111 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-28T15:50:43.178 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-28T15:50:43.233 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-28T15:50:43.234 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/a1effab339b46693621f83d849345b020354fac9c91be6d17236ad20a0b73f09/merged 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f735f5f96a26be5db08a5931a68c05aed51050f6c123368dafa5ba27a44e125d/merged 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/160d8dc4b8fb4bcf5af9fc4089b7229eb3d34323a38f3c113b04638349a8f79f/merged 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/8ac9f704d3276e021ac8932b9e84d915e21d94d036c118999e87184beaece252/merged 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/29eda3d6a669f7553aaf2f04a876c6cd8ac7eab1139fd8f09451b9b073915090/merged 2024-03-28T15:50:43.235 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d3a25d8d50b398d6a3010de5ac3c6f2cbfeb96c573a23f8cbef1202a5c5c5b21/merged 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/c60356adfbee0fa15457daabb126aad0d641c7ee29776e1f340c6185b75cbe18/merged 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/bcb73bbb3fd8dc95cc1280d14fc486873d6e2e956ea44496aec984de587d007f/merged 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/881a3aab06f4a8c61a81d9c5b0d03b40a5ad0322a9f4a30e41443b15c0e13940/merged 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ec950c9912ec25530a3bad89eb7bf806aba9718918aabbc5b4c30c425f53ba76/merged 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:50:43.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.236 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-28T15:50:43.287 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-28T15:50:43.287 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-28T15:50:43.329 DEBUG:teuthology.orchestra.run.smithi023:> sudo modprobe fuse 2024-03-28T15:50:43.356 DEBUG:teuthology.orchestra.run.smithi023:> cat /proc/self/mounts | awk '{print $2}' 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/proc 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/sys 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/dev 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/security 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/shm 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/pts 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/run 2024-03-28T15:50:43.411 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/cgroup 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/pstore 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/bpf 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/ 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/selinux 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/hugepages 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/dev/mqueue 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/debug 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/tracing 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/fs/fuse/connections 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/sys/kernel/config 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-sysctl.service 2024-03-28T15:50:43.412 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/nfs/rpc_pipefs 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/1000 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/ceph 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/a1effab339b46693621f83d849345b020354fac9c91be6d17236ad20a0b73f09/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/f735f5f96a26be5db08a5931a68c05aed51050f6c123368dafa5ba27a44e125d/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/run/user/0 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/proc/sys/fs/binfmt_misc 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/160d8dc4b8fb4bcf5af9fc4089b7229eb3d34323a38f3c113b04638349a8f79f/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/8ac9f704d3276e021ac8932b9e84d915e21d94d036c118999e87184beaece252/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/29eda3d6a669f7553aaf2f04a876c6cd8ac7eab1139fd8f09451b9b073915090/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/d3a25d8d50b398d6a3010de5ac3c6f2cbfeb96c573a23f8cbef1202a5c5c5b21/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/c60356adfbee0fa15457daabb126aad0d641c7ee29776e1f340c6185b75cbe18/merged 2024-03-28T15:50:43.413 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/bcb73bbb3fd8dc95cc1280d14fc486873d6e2e956ea44496aec984de587d007f/merged 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/881a3aab06f4a8c61a81d9c5b0d03b40a5ad0322a9f4a30e41443b15c0e13940/merged 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run.smithi023.stdout:/var/lib/containers/storage/overlay/ec950c9912ec25530a3bad89eb7bf806aba9718918aabbc5b4c30c425f53ba76/merged 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run.smithi023.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:50:43.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.414 DEBUG:teuthology.orchestra.run.smithi023:> ls /sys/fs/fuse/connections 2024-03-28T15:50:43.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:2024-03-28T15:50:43.431+0000 7f622be38480 -1 init, newargv = 0x556337efbe20 newargc=15 2024-03-28T15:50:43.431 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75645]: starting ceph client 2024-03-28T15:50:43.436 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75645]: starting fuse 2024-03-28T15:50:43.446 INFO:teuthology.orchestra.run.smithi023.stdout:119 2024-03-28T15:50:43.446 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo stdin-killer -- python3 -c ' 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> import glob 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> import re 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> import os 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> import subprocess 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> def _find_admin_socket(client_name): 2024-03-28T15:50:43.446 DEBUG:teuthology.orchestra.run.smithi023:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> files = glob.glob(asok_path) 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> # Given a non-glob path, it better be there 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> if "*" not in asok_path: 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> assert(len(files) == 1) 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> return files[0] 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> for f in files: 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> contents = proc_f.read() 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> if mountpoint in contents: 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> return f 2024-03-28T15:50:43.447 DEBUG:teuthology.orchestra.run.smithi023:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-28T15:50:43.448 DEBUG:teuthology.orchestra.run.smithi023:> 2024-03-28T15:50:43.448 DEBUG:teuthology.orchestra.run.smithi023:> print(_find_admin_socket("client.admin")) 2024-03-28T15:50:43.448 DEBUG:teuthology.orchestra.run.smithi023:> ' 2024-03-28T15:50:43.567 INFO:teuthology.orchestra.run.smithi023.stdout:/var/run/ceph/ceph-client.admin.75645.asok 2024-03-28T15:50:43.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-28T15:50:43.578 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.75645.asok 2024-03-28T15:50:43.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.578 DEBUG:teuthology.orchestra.run.smithi023:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.75645.asok status 2024-03-28T15:50:43.716 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:50:43.716 INFO:teuthology.orchestra.run.smithi023.stdout: "metadata": { 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_sha1": "3be9a5c9bc793e11e2800a8c0c696e8b46742033", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "entity_id": "admin", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": "smithi023", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "pid": "75645", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "root": "/" 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_count": 0, 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "dentry_pinned_count": 0, 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14772, 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "inst": { 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": { 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "client", 2024-03-28T15:50:43.717 INFO:teuthology.orchestra.run.smithi023.stdout: "num": 14772 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1879209041 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": { 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "192.168.0.1:0", 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1879209041 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T15:50:43.718 INFO:teuthology.orchestra.run.smithi023.stdout: "inst_str": "client.14772 192.168.0.1:0/1879209041", 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "addr_str": "192.168.0.1:0/1879209041", 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "inode_count": 1, 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_epoch": 17, 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch": 90, 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_epoch_barrier": 0, 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "blocklisted": false, 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs" 2024-03-28T15:50:43.719 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:50:43.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:43 smithi023 ceph-mon[32448]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 295 B/s wr, 0 op/s 2024-03-28T15:50:43.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.725 DEBUG:teuthology.orchestra.run.smithi023:> stat --file-system '--printf=%T 2024-03-28T15:50:43.725 DEBUG:teuthology.orchestra.run.smithi023:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:43.740 INFO:teuthology.orchestra.run.smithi023.stdout:fuseblk 2024-03-28T15:50:43.740 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:43.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.741 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:43.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:43.808 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-28T15:50:43.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:43 smithi117 ceph-mon[36002]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 295 B/s wr, 0 op/s 2024-03-28T15:50:43.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:43 smithi120 ceph-mon[36826]: pgmap v294: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 295 B/s wr, 0 op/s 2024-03-28T15:50:43.892 INFO:teuthology.orchestra.run.smithi023.stdout:# file: . 2024-03-28T15:50:43.892 INFO:teuthology.orchestra.run.smithi023.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-28T15:50:43.892 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:43.893 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:50:43.893 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:50:43.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- 172.21.15.23:0/423451378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae01028c0 msgr2=0x7fdae0102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 --2- 172.21.15.23:0/423451378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae01028c0 0x7fdae0102d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdad4009900 tx=0x7fdad402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- 172.21.15.23:0/423451378 shutdown_connections 2024-03-28T15:50:44.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 --2- 172.21.15.23:0/423451378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae0103280 0x7fdae0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 --2- 172.21.15.23:0/423451378 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae01028c0 0x7fdae0102d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 --2- 172.21.15.23:0/423451378 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01016d0 0x7fdae0101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- 172.21.15.23:0/423451378 >> 172.21.15.23:0/423451378 conn(0x7fdae00fcc40 msgr2=0x7fdae00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- 172.21.15.23:0/423451378 shutdown_connections 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- 172.21.15.23:0/423451378 wait complete. 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 Processor -- start 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.092+0000 7fdae649a640 1 -- start start 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 0x7fdae0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01028c0 0x7fdae0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae010d330 con 0x7fdae0103280 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae010d1b0 con 0x7fdae01016d0 2024-03-28T15:50:44.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae649a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae010d4b0 con 0x7fdae01028c0 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43604/0 (socket says 172.21.15.23:43604) 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 -- 172.21.15.23:0/182002721 learned_addr learned my addr 172.21.15.23:0/182002721 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdadffff640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 0x7fdae0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdadf7fe640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01028c0 0x7fdae0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 -- 172.21.15.23:0/182002721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01028c0 msgr2=0x7fdae0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01028c0 0x7fdae0197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 -- 172.21.15.23:0/182002721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 msgr2=0x7fdae0197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 0x7fdae0197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 -- 172.21.15.23:0/182002721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae019f560 con 0x7fdae0103280 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdadffff640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 0x7fdae0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:44.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.093+0000 7fdae4a37640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdad800eb10 tx=0x7fdad800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.094+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8004b20 con 0x7fdae0103280 2024-03-28T15:50:44.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.094+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae019f850 con 0x7fdae0103280 2024-03-28T15:50:44.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.094+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdad8004cc0 con 0x7fdae0103280 2024-03-28T15:50:44.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.094+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad8010760 con 0x7fdae0103280 2024-03-28T15:50:44.097 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.094+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae019fd20 con 0x7fdae0103280 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.095+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae0108990 con 0x7fdae0103280 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.096+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdad8005020 con 0x7fdae0103280 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.096+0000 7fdadd7fa640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 0x7fdac80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.096+0000 7fdadffff640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 0x7fdac80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.096+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdad80cf210 con 0x7fdae0103280 2024-03-28T15:50:44.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.097+0000 7fdadffff640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 0x7fdac80a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdae0102720 tx=0x7fdad0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.101+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdad809c070 con 0x7fdae0103280 2024-03-28T15:50:44.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.285+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdae002d070 con 0x7fdae0103280 2024-03-28T15:50:44.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.287+0000 7fdadd7fa640 1 -- 172.21.15.23:0/182002721 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7fdad809b970 con 0x7fdae0103280 2024-03-28T15:50:44.287 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:44.287 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 msgr2=0x7fdac80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 0x7fdac80a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdae0102720 tx=0x7fdad0005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 msgr2=0x7fdae019eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fdad800eb10 tx=0x7fdad800c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 shutdown_connections 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdac80a2800 0x7fdac80a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdae0103280 0x7fdae019eef0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdae01028c0 0x7fdae0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 --2- 172.21.15.23:0/182002721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdae01016d0 0x7fdae0197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 >> 172.21.15.23:0/182002721 conn(0x7fdae00fcc40 msgr2=0x7fdae0105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 shutdown_connections 2024-03-28T15:50:44.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.289+0000 7fdae649a640 1 -- 172.21.15.23:0/182002721 wait complete. 2024-03-28T15:50:44.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:50:44.302 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- 172.21.15.23:0/3364622831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 msgr2=0x7fb6d01009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 --2- 172.21.15.23:0/3364622831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d01009c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6c0009900 tx=0x7fb6c002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- 172.21.15.23:0/3364622831 shutdown_connections 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 --2- 172.21.15.23:0/3364622831 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 --2- 172.21.15.23:0/3364622831 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 0x7fb6d0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 --2- 172.21.15.23:0/3364622831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d01009c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- 172.21.15.23:0/3364622831 >> 172.21.15.23:0/3364622831 conn(0x7fb6d00fbb30 msgr2=0x7fb6d00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- 172.21.15.23:0/3364622831 shutdown_connections 2024-03-28T15:50:44.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- 172.21.15.23:0/3364622831 wait complete. 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 Processor -- start 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.409+0000 7fb6d684d640 1 -- start start 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d0104530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 0x7fb6d0102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d010c260 con 0x7fb6d0102150 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d010c0e0 con 0x7fb6d01005a0 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d010c3e0 con 0x7fb6d0101790 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6cffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d0104530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43624/0 (socket says 172.21.15.23:43624) 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 -- 172.21.15.23:0/2084908743 learned_addr learned my addr 172.21.15.23:0/2084908743 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:44.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6cf7fe640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 0x7fb6d0102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 -- 172.21.15.23:0/2084908743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 msgr2=0x7fb6d0102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 0x7fb6d0102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 -- 172.21.15.23:0/2084908743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 msgr2=0x7fb6d0104530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d0104530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 -- 172.21.15.23:0/2084908743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d0103940 con 0x7fb6d0102150 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d4dea640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800efd0 tx=0x7fb6c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c8003b70 con 0x7fb6d0102150 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6c8003d10 con 0x7fb6d0102150 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d01a1bd0 con 0x7fb6d0102150 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6c80109f0 con 0x7fb6d0102150 2024-03-28T15:50:44.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.410+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d01a20d0 con 0x7fb6d0102150 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.412+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d01a2300 con 0x7fb6d0102150 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.413+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb6c8010b90 con 0x7fb6d0102150 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.413+0000 7fb6cd7fa640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 0x7fb6b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.414+0000 7fb6cffff640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 0x7fb6b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.414+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb6c80cf4d0 con 0x7fb6d0102150 2024-03-28T15:50:44.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.414+0000 7fb6cffff640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 0x7fb6b80a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c170 tx=0x7fb6c002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.417+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb6c809c330 con 0x7fb6d0102150 2024-03-28T15:50:44.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.604+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb6d01a24d0 con 0x7fb6d0102150 2024-03-28T15:50:44.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.605+0000 7fb6cd7fa640 1 -- 172.21.15.23:0/2084908743 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb6c809bc30 con 0x7fb6d0102150 2024-03-28T15:50:44.605 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.606+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 msgr2=0x7fb6b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.606+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 0x7fb6b80a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb6c002c170 tx=0x7fb6c002df90 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.606+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 msgr2=0x7fb6d01030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.606+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800efd0 tx=0x7fb6c800c6a0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 shutdown_connections 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6b80a2800 0x7fb6b80a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6d0102150 0x7fb6d01030b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6d0101790 0x7fb6d0102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 --2- 172.21.15.23:0/2084908743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6d01005a0 0x7fb6d0104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 >> 172.21.15.23:0/2084908743 conn(0x7fb6d00fbb30 msgr2=0x7fb6d00fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 shutdown_connections 2024-03-28T15:50:44.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.607+0000 7fb6d684d640 1 -- 172.21.15.23:0/2084908743 wait complete. 2024-03-28T15:50:44.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:44 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/182002721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:44.617 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 -- 172.21.15.23:0/1848719796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa81005a0 msgr2=0x7effa81009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 --2- 172.21.15.23:0/1848719796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa81005a0 0x7effa81009c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7effa0009900 tx=0x7effa002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 -- 172.21.15.23:0/1848719796 shutdown_connections 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 --2- 172.21.15.23:0/1848719796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8102150 0x7effa81089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 --2- 172.21.15.23:0/1848719796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8101790 0x7effa8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 --2- 172.21.15.23:0/1848719796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa81005a0 0x7effa81009c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 -- 172.21.15.23:0/1848719796 >> 172.21.15.23:0/1848719796 conn(0x7effa80fbb30 msgr2=0x7effa80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 -- 172.21.15.23:0/1848719796 shutdown_connections 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.722+0000 7effb0177640 1 -- 172.21.15.23:0/1848719796 wait complete. 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 Processor -- start 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 -- start start 2024-03-28T15:50:44.725 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa81005a0 0x7effa8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102150 0x7effa819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d3c0 con 0x7effa8101790 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d240 con 0x7effa8102150 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effb0177640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d540 con 0x7effa81005a0 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effad712640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effad712640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43654/0 (socket says 172.21.15.23:43654) 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effad712640 1 -- 172.21.15.23:0/3422456418 learned_addr learned my addr 172.21.15.23:0/3422456418 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effae714640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102150 0x7effa819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effadf13640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa81005a0 0x7effa8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.723+0000 7effad712640 1 -- 172.21.15.23:0/3422456418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa81005a0 msgr2=0x7effa8197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.726 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effad712640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa81005a0 0x7effa8197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effad712640 1 -- 172.21.15.23:0/3422456418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102150 msgr2=0x7effa819f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effad712640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102150 0x7effa819f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effad712640 1 -- 172.21.15.23:0/3422456418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa819f6a0 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effad712640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e9f0 tx=0x7eff9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c00cd80 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff9c010910 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa819f990 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c018990 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.724+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa819fee0 con 0x7effa8101790 2024-03-28T15:50:44.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.725+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa81078c0 con 0x7effa8101790 2024-03-28T15:50:44.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.726+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7eff9c010ac0 con 0x7effa8101790 2024-03-28T15:50:44.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.726+0000 7effa6ffd640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:44.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.726+0000 7effadf13640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:44.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.727+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7eff9c0cf080 con 0x7effa8101790 2024-03-28T15:50:44.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.727+0000 7effadf13640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7effa002c170 tx=0x7effa002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:44.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.731+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff9c09bee0 con 0x7effa8101790 2024-03-28T15:50:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:44 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/182002721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:44 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/182002721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:44.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.915+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7effa80622d0 con 0x7effa8101790 2024-03-28T15:50:44.916 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.915+0000 7effa6ffd640 1 -- 172.21.15.23:0/3422456418 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7eff9c09b7e0 con 0x7effa8101790 2024-03-28T15:50:44.916 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 msgr2=0x7eff900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7effa002c170 tx=0x7effa002df90 comp rx=0 tx=0).stop 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 msgr2=0x7effa8197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e9f0 tx=0x7eff9c00eeb0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 shutdown_connections 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102150 0x7effa819f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.917+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8101790 0x7effa8197fc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.918+0000 7effb0177640 1 --2- 172.21.15.23:0/3422456418 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa81005a0 0x7effa8197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:44.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.918+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 >> 172.21.15.23:0/3422456418 conn(0x7effa80fbb30 msgr2=0x7effa8104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:44.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.918+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 shutdown_connections 2024-03-28T15:50:44.919 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:44.918+0000 7effb0177640 1 -- 172.21.15.23:0/3422456418 wait complete. 2024-03-28T15:50:44.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 -- 172.21.15.23:0/2337568055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 msgr2=0x7fb9941009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 --2- 172.21.15.23:0/2337568055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941009e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb990009900 tx=0x7fb99002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 -- 172.21.15.23:0/2337568055 shutdown_connections 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 --2- 172.21.15.23:0/2337568055 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb994108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 --2- 172.21.15.23:0/2337568055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 0x7fb994101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 --2- 172.21.15.23:0/2337568055 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941009e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 -- 172.21.15.23:0/2337568055 >> 172.21.15.23:0/2337568055 conn(0x7fb9940fbb50 msgr2=0x7fb9940fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 -- 172.21.15.23:0/2337568055 shutdown_connections 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.033+0000 7fb99b492640 1 -- 172.21.15.23:0/2337568055 wait complete. 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 Processor -- start 2024-03-28T15:50:45.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 -- start start 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 0x7fb994197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb99419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99410c230 con 0x7fb9941005c0 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99410c0b0 con 0x7fb9941017b0 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99b492640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99410c3b0 con 0x7fb994102170 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43668/0 (socket says 172.21.15.23:43668) 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 -- 172.21.15.23:0/2280316040 learned_addr learned my addr 172.21.15.23:0/2280316040 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb998a2d640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 0x7fb994197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 -- 172.21.15.23:0/2280316040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 msgr2=0x7fb99419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb999a2f640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb99419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb99419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 -- 172.21.15.23:0/2280316040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 msgr2=0x7fb994197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 0x7fb994197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb99922e640 1 -- 172.21.15.23:0/2280316040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99419f610 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.034+0000 7fb999a2f640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb99419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb99922e640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb990000c00 tx=0x7fb990005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990041070 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9900044d0 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99419f8a0 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb990034440 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.035+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99419fd90 con 0x7fb9941005c0 2024-03-28T15:50:45.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.036+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9941a0050 con 0x7fb9941005c0 2024-03-28T15:50:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.037+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb990044070 con 0x7fb9941005c0 2024-03-28T15:50:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.037+0000 7fb98a7fc640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 0x7fb9800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.038+0000 7fb998a2d640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 0x7fb9800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.038+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb9900f0a60 con 0x7fb9941005c0 2024-03-28T15:50:45.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.038+0000 7fb998a2d640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 0x7fb9800a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb9840095f0 tx=0x7fb984005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.041+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9900bd8c0 con 0x7fb9941005c0 2024-03-28T15:50:45.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.221+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb9941a03e0 con 0x7fb9941005c0 2024-03-28T15:50:45.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.222+0000 7fb98a7fc640 1 -- 172.21.15.23:0/2280316040 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb9900bd1c0 con 0x7fb9941005c0 2024-03-28T15:50:45.222 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:50:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 msgr2=0x7fb9800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 0x7fb9800a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb9840095f0 tx=0x7fb984005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 msgr2=0x7fb9941979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb990000c00 tx=0x7fb990005d50 comp rx=0 tx=0).stop 2024-03-28T15:50:45.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 shutdown_connections 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb9800a2800 0x7fb9800a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb994102170 0x7fb99419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9941017b0 0x7fb994197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 --2- 172.21.15.23:0/2280316040 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9941005c0 0x7fb9941979f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 >> 172.21.15.23:0/2280316040 conn(0x7fb9940fbb50 msgr2=0x7fb9940fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 shutdown_connections 2024-03-28T15:50:45.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.224+0000 7fb99b492640 1 -- 172.21.15.23:0/2280316040 wait complete. 2024-03-28T15:50:45.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:50:45.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- 172.21.15.23:0/2073318238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 msgr2=0x7fbc00106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 --2- 172.21.15.23:0/2073318238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00106d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4009900 tx=0x7fbbf402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- 172.21.15.23:0/2073318238 shutdown_connections 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 --2- 172.21.15.23:0/2073318238 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 0x7fbc001096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 --2- 172.21.15.23:0/2073318238 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00106d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 --2- 172.21.15.23:0/2073318238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc00104410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- 172.21.15.23:0/2073318238 >> 172.21.15.23:0/2073318238 conn(0x7fbc000fba10 msgr2=0x7fbc000fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- 172.21.15.23:0/2073318238 shutdown_connections 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- 172.21.15.23:0/2073318238 wait complete. 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 Processor -- start 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.340+0000 7fbc0539d640 1 -- start start 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc001978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 0x7fbc0019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0010d230 con 0x7fbc00104950 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0010d0b0 con 0x7fbc00102020 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc0539d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0010d3b0 con 0x7fbc00107280 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43686/0 (socket says 172.21.15.23:43686) 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 -- 172.21.15.23:0/3705736311 learned_addr learned my addr 172.21.15.23:0/3705736311 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbfffff640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc001978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 -- 172.21.15.23:0/3705736311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 msgr2=0x7fbc0019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbc04b9c640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 0x7fbc0019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 0x7fbc0019ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 -- 172.21.15.23:0/3705736311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 msgr2=0x7fbc001978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc001978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 -- 172.21.15.23:0/3705736311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc0019f4e0 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbfffff640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc001978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbff7fe640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4000c00 tx=0x7fbbf4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.341+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4041070 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.342+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbf40044d0 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.342+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0019f770 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.342+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbf4034440 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.342+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0019fc60 con 0x7fbc00104950 2024-03-28T15:50:45.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.343+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc000fd500 con 0x7fbc00104950 2024-03-28T15:50:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.344+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbbf4044070 con 0x7fbc00104950 2024-03-28T15:50:45.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.344+0000 7fbbfd7fa640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 0x7fbbe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.344+0000 7fbbfffff640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 0x7fbbe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.344+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbbf40f0840 con 0x7fbc00104950 2024-03-28T15:50:45.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.345+0000 7fbbfffff640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 0x7fbbe80a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009fc0 tx=0x7fbbf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.348 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.349+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbbf40bd6a0 con 0x7fbc00104950 2024-03-28T15:50:45.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.533+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbc000622a0 con 0x7fbc00104950 2024-03-28T15:50:45.533 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.533+0000 7fbbfd7fa640 1 -- 172.21.15.23:0/3705736311 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbbf40bcfa0 con 0x7fbc00104950 2024-03-28T15:50:45.533 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:50:45.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 msgr2=0x7fbbe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 0x7fbbe80a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbf0009fc0 tx=0x7fbbf0005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 msgr2=0x7fbc00197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fbbf4000c00 tx=0x7fbbf4005d50 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 shutdown_connections 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbbe80a2800 0x7fbbe80a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbc00107280 0x7fbc0019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbc00104950 0x7fbc00197e00 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 --2- 172.21.15.23:0/3705736311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc00102020 0x7fbc001978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 >> 172.21.15.23:0/3705736311 conn(0x7fbc000fba10 msgr2=0x7fbc00102960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 shutdown_connections 2024-03-28T15:50:45.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.535+0000 7fbc0539d640 1 -- 172.21.15.23:0/3705736311 wait complete. 2024-03-28T15:50:45.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:45 smithi023 ceph-mon[32448]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T15:50:45.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2084908743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3422456418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2280316040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.546 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T15:50:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/778785247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88101790 msgr2=0x7f1d88101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/778785247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88101790 0x7f1d88101c10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009900 tx=0x7f1d7002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:45.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/778785247 shutdown_connections 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/778785247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d88102150 0x7f1d881089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/778785247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88101790 0x7f1d88101c10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/778785247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d881005a0 0x7f1d881009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/778785247 >> 172.21.15.23:0/778785247 conn(0x7f1d880fbb30 msgr2=0x7f1d880fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/778785247 shutdown_connections 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.649+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/778785247 wait complete. 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 Processor -- start 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 -- start start 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d881005a0 0x7f1d881979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 0x7f1d88197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8810d340 con 0x7f1d88102150 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8810d1c0 con 0x7f1d88101790 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8d1b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8810d4c0 con 0x7f1d881005a0 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.650+0000 7f1d8759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8659c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 0x7f1d88197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43714/0 (socket says 172.21.15.23:43714) 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 -- 172.21.15.23:0/3559033983 learned_addr learned my addr 172.21.15.23:0/3559033983 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d86d9d640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d881005a0 0x7f1d881979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 -- 172.21.15.23:0/3559033983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d881005a0 msgr2=0x7f1d881979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d881005a0 0x7f1d881979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 -- 172.21.15.23:0/3559033983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 msgr2=0x7f1d88197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 0x7f1d88197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 -- 172.21.15.23:0/3559033983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8819f5e0 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8659c640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 0x7f1d88197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8759e640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800eaa0 tx=0x7f1d7800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7800cdb0 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8819f8d0 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d78005020 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d780116d0 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.651+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8819fe40 con 0x7f1d88102150 2024-03-28T15:50:45.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.652+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d881078e0 con 0x7f1d88102150 2024-03-28T15:50:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.653+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1d780054d0 con 0x7f1d88102150 2024-03-28T15:50:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.654+0000 7f1d7ffff640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 0x7f1d740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.654+0000 7f1d86d9d640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 0x7f1d740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:45.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.654+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1d780cf190 con 0x7f1d88102150 2024-03-28T15:50:45.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.655+0000 7f1d86d9d640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 0x7f1d740a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d881015f0 tx=0x7f1d80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:45.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.658+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d78016030 con 0x7f1d88102150 2024-03-28T15:50:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:45 smithi117 ceph-mon[36002]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T15:50:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:45 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2084908743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:45 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3422456418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:45 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2280316040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:45 smithi120 ceph-mon[36826]: pgmap v295: 129 pgs: 129 active+clean; 579 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-28T15:50:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:45 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2084908743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:45 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3422456418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:45 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2280316040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:45.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.854+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1d8802d070 con 0x7f1d88102150 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.855+0000 7f1d7ffff640 1 -- 172.21.15.23:0/3559033983 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+591 (secure 0 0 0) 0x7f1d7809b9a0 con 0x7f1d88102150 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2505463471 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3395482026 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:45.855 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2095255310 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:45.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:45.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1534223304 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:45.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/611179097 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:45.856 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1777550602 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:45.856 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 msgr2=0x7f1d740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 0x7f1d740a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1d881015f0 tx=0x7f1d80005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 msgr2=0x7f1d8819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1d7800eaa0 tx=0x7f1d7800ef60 comp rx=0 tx=0).stop 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 shutdown_connections 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d740a2800 0x7f1d740a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d88102150 0x7f1d8819ef70 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d88101790 0x7f1d88197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 --2- 172.21.15.23:0/3559033983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d881005a0 0x7f1d881979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 >> 172.21.15.23:0/3559033983 conn(0x7f1d880fbb30 msgr2=0x7f1d88104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 shutdown_connections 2024-03-28T15:50:45.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:45.857+0000 7f1d8d1b2640 1 -- 172.21.15.23:0/3559033983 wait complete. 2024-03-28T15:50:45.868 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:50:45.868 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:50:45.883 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:46.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 -- 172.21.15.23:0/3523553557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 msgr2=0x7f8a5c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/3523553557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c1045b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009900 tx=0x7f8a5002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:46.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 -- 172.21.15.23:0/3523553557 shutdown_connections 2024-03-28T15:50:46.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/3523553557 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 0x7f8a5c109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/3523553557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 0x7f8a5c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/3523553557 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c1045b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 -- 172.21.15.23:0/3523553557 >> 172.21.15.23:0/3523553557 conn(0x7f8a5c0fbbd0 msgr2=0x7f8a5c0fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 -- 172.21.15.23:0/3523553557 shutdown_connections 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.027+0000 7f8a62ff3640 1 -- 172.21.15.23:0/3523553557 wait complete. 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 Processor -- start 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 -- start start 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 0x7f8a5c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 0x7f8a5c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10d360 con 0x7f8a5c1021c0 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10d1e0 con 0x7f8a5c104af0 2024-03-28T15:50:46.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a62ff3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c10d4e0 con 0x7f8a5c107420 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43728/0 (socket says 172.21.15.23:43728) 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 -- 172.21.15.23:0/852082507 learned_addr learned my addr 172.21.15.23:0/852082507 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a5bfff640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 0x7f8a5c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a61590640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 0x7f8a5c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 -- 172.21.15.23:0/852082507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 msgr2=0x7f8a5c19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 0x7f8a5c19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 -- 172.21.15.23:0/852082507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 msgr2=0x7f8a5c197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 0x7f8a5c197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 -- 172.21.15.23:0/852082507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c19f640 con 0x7f8a5c1021c0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a60d8f640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009230 tx=0x7f8a50005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.028+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50041070 con 0x7f8a5c1021c0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.029+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a500044d0 con 0x7f8a5c1021c0 2024-03-28T15:50:46.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.029+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c19f8d0 con 0x7f8a5c1021c0 2024-03-28T15:50:46.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.029+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a50034440 con 0x7f8a5c1021c0 2024-03-28T15:50:46.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.029+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c19fdc0 con 0x7f8a5c1021c0 2024-03-28T15:50:46.032 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.030+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c0fd6c0 con 0x7f8a5c1021c0 2024-03-28T15:50:46.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.031+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8a50044070 con 0x7f8a5c1021c0 2024-03-28T15:50:46.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.031+0000 7f8a59ffb640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 0x7f8a4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.031+0000 7f8a5bfff640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 0x7f8a4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.031+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8a500f0870 con 0x7f8a5c1021c0 2024-03-28T15:50:46.033 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.032+0000 7f8a5bfff640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 0x7f8a4c0a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a480095f0 tx=0x7f8a48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.035+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8a500bd080 con 0x7f8a5c1021c0 2024-03-28T15:50:46.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.220+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a5c0622a0 con 0x7f8a5c1021c0 2024-03-28T15:50:46.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.221+0000 7f8a59ffb640 1 -- 172.21.15.23:0/852082507 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7f8a500c1120 con 0x7f8a5c1021c0 2024-03-28T15:50:46.221 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:46.222 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 msgr2=0x7f8a4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 0x7f8a4c0a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8a480095f0 tx=0x7f8a48005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 msgr2=0x7f8a5c197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a50009230 tx=0x7f8a50005d50 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 shutdown_connections 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8a4c0a2800 0x7f8a4c0a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8a5c107420 0x7f8a5c19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8a5c104af0 0x7f8a5c197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 --2- 172.21.15.23:0/852082507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8a5c1021c0 0x7f8a5c197a20 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 >> 172.21.15.23:0/852082507 conn(0x7f8a5c0fbbd0 msgr2=0x7f8a5c102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 shutdown_connections 2024-03-28T15:50:46.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.223+0000 7f8a62ff3640 1 -- 172.21.15.23:0/852082507 wait complete. 2024-03-28T15:50:46.235 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:50:46.235 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 -- 172.21.15.23:0/111032156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01016d0 msgr2=0x7f30f0101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/111032156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01016d0 0x7f30f0101af0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f30e8009900 tx=0x7f30e802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 -- 172.21.15.23:0/111032156 shutdown_connections 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/111032156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/111032156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01028c0 0x7f30f0102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/111032156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01016d0 0x7f30f0101af0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.340+0000 7f30f7f7e640 1 -- 172.21.15.23:0/111032156 >> 172.21.15.23:0/111032156 conn(0x7f30f00fcc40 msgr2=0x7f30f00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- 172.21.15.23:0/111032156 shutdown_connections 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- 172.21.15.23:0/111032156 wait complete. 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 Processor -- start 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- start start 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01016d0 0x7f30f0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f010d350 con 0x7f30f01028c0 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f010d1d0 con 0x7f30f01016d0 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f7f7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f010d4d0 con 0x7f30f0103280 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f5519640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f5519640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43740/0 (socket says 172.21.15.23:43740) 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f5519640 1 -- 172.21.15.23:0/2642183536 learned_addr learned my addr 172.21.15.23:0/2642183536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:46.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f5d1a640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01016d0 0x7f30f0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.341+0000 7f30f651b640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 -- 172.21.15.23:0/2642183536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 msgr2=0x7f30f0073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 -- 172.21.15.23:0/2642183536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01016d0 msgr2=0x7f30f0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01016d0 0x7f30f0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 -- 172.21.15.23:0/2642183536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f0074860 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f651b640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f5519640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30e400ea60 tx=0x7f30e400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e400cde0 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30e40035f0 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f01a1b40 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4010670 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.342+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f01a2090 con 0x7f30f01028c0 2024-03-28T15:50:46.345 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.343+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f0108a40 con 0x7f30f01028c0 2024-03-28T15:50:46.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.344+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f30e40037a0 con 0x7f30f01028c0 2024-03-28T15:50:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.345+0000 7f30eeffd640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 0x7f30d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.345+0000 7f30f5d1a640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 0x7f30d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.345+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f30e40cf090 con 0x7f30f01028c0 2024-03-28T15:50:46.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.345+0000 7f30f5d1a640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 0x7f30d80a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30e8000c00 tx=0x7f30e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.349+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f30e409bef0 con 0x7f30f01028c0 2024-03-28T15:50:46.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.533+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30f00622d0 con 0x7f30f01028c0 2024-03-28T15:50:46.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.534+0000 7f30eeffd640 1 -- 172.21.15.23:0/2642183536 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30e409b7f0 con 0x7f30f01028c0 2024-03-28T15:50:46.534 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 msgr2=0x7f30d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 0x7f30d80a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f30e8000c00 tx=0x7f30e8005fd0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 msgr2=0x7f30f0073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f30e400ea60 tx=0x7f30e400ef20 comp rx=0 tx=0).stop 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 shutdown_connections 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f30d80a2800 0x7f30d80a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.536 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f30f0103280 0x7f30f0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f30f01028c0 0x7f30f0073a90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 --2- 172.21.15.23:0/2642183536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30f01016d0 0x7f30f0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 >> 172.21.15.23:0/2642183536 conn(0x7f30f00fcc40 msgr2=0x7f30f0105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 shutdown_connections 2024-03-28T15:50:46.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.536+0000 7f30f7f7e640 1 -- 172.21.15.23:0/2642183536 wait complete. 2024-03-28T15:50:46.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:46 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3705736311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:46.545 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:46 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3559033983' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:46.546 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:46 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/852082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:46.547 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:50:46.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/3564798921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81017a0 msgr2=0x7fd0f8101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/3564798921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81017a0 0x7fd0f8101c20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec009900 tx=0x7fd0ec02bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/3564798921 shutdown_connections 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/3564798921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f8102160 0x7fd0f81089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/3564798921 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81017a0 0x7fd0f8101c20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/3564798921 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f81005b0 0x7fd0f81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/3564798921 >> 172.21.15.23:0/3564798921 conn(0x7fd0f80fbb40 msgr2=0x7fd0f80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/3564798921 shutdown_connections 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.652+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/3564798921 wait complete. 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 Processor -- start 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 -- start start 2024-03-28T15:50:46.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f81017a0 0x7fd0f8197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 0x7fd0f819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810c210 con 0x7fd0f81005b0 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810c090 con 0x7fd0f81017a0 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0fe0ea640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f810c390 con 0x7fd0f8102160 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0f77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43762/0 (socket says 172.21.15.23:43762) 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0f77fe640 1 -- 172.21.15.23:0/4189854976 learned_addr learned my addr 172.21.15.23:0/4189854976 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0f6ffd640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f81017a0 0x7fd0f8197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.653+0000 7fd0f7fff640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 0x7fd0f819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 -- 172.21.15.23:0/4189854976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 msgr2=0x7fd0f819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.656 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 0x7fd0f819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 -- 172.21.15.23:0/4189854976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f81017a0 msgr2=0x7fd0f8197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f81017a0 0x7fd0f8197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 -- 172.21.15.23:0/4189854976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f819f650 con 0x7fd0f81005b0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f7fff640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 0x7fd0f819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f77fe640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d770 tx=0x7fd0e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8017070 con 0x7fd0f81005b0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f819f940 con 0x7fd0f81005b0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0e8009c30 con 0x7fd0f81005b0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8003dd0 con 0x7fd0f81005b0 2024-03-28T15:50:46.657 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.654+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f819feb0 con 0x7fd0f81005b0 2024-03-28T15:50:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.655+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f81a00e0 con 0x7fd0f81005b0 2024-03-28T15:50:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.656+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd0e8004000 con 0x7fd0f81005b0 2024-03-28T15:50:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.657+0000 7fd0f4ff9640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 0x7fd0e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.657+0000 7fd0f6ffd640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 0x7fd0e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.657+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd0e80ceab0 con 0x7fd0f81005b0 2024-03-28T15:50:46.659 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.657+0000 7fd0f6ffd640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 0x7fd0e00a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02c7b0 tx=0x7fd0ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.660 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.660+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0e8009de0 con 0x7fd0f81005b0 2024-03-28T15:50:46.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:46 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3705736311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:46.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:46 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3559033983' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:46.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:46 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/852082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:46 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3705736311' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:46 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3559033983' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:46 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/852082507' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:46.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.841+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd0f81a02b0 con 0x7fd0f81005b0 2024-03-28T15:50:46.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.842+0000 7fd0f4ff9640 1 -- 172.21.15.23:0/4189854976 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd0e809b8b0 con 0x7fd0f81005b0 2024-03-28T15:50:46.842 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.843+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 msgr2=0x7fd0e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.843+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 0x7fd0e00a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec02c7b0 tx=0x7fd0ec005fd0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 msgr2=0x7fd0f8197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800d770 tx=0x7fd0e800dc30 comp rx=0 tx=0).stop 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 shutdown_connections 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0e00a2800 0x7fd0e00a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0f8102160 0x7fd0f819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0f81017a0 0x7fd0f8197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 --2- 172.21.15.23:0/4189854976 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd0f81005b0 0x7fd0f8197a80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 >> 172.21.15.23:0/4189854976 conn(0x7fd0f80fbb40 msgr2=0x7fd0f8069c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 shutdown_connections 2024-03-28T15:50:46.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.844+0000 7fd0fe0ea640 1 -- 172.21.15.23:0/4189854976 wait complete. 2024-03-28T15:50:46.855 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:50:46.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 -- 172.21.15.23:0/1899811246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 msgr2=0x7f5768108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 --2- 172.21.15.23:0/1899811246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f5768108a00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5760009ca0 tx=0x7f576002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 -- 172.21.15.23:0/1899811246 shutdown_connections 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 --2- 172.21.15.23:0/1899811246 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f5768108a00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 --2- 172.21.15.23:0/1899811246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 0x7f5768101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 --2- 172.21.15.23:0/1899811246 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 0x7f57681009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 -- 172.21.15.23:0/1899811246 >> 172.21.15.23:0/1899811246 conn(0x7f57680fbb50 msgr2=0x7f57680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 -- 172.21.15.23:0/1899811246 shutdown_connections 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.960+0000 7f576ce59640 1 -- 172.21.15.23:0/1899811246 wait complete. 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 Processor -- start 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 -- start start 2024-03-28T15:50:46.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 0x7f5768197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 0x7f5768197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576810c220 con 0x7f5768102170 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576810c0a0 con 0x7f57681017b0 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576ce59640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576810c3a0 con 0x7f57681005c0 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43786/0 (socket says 172.21.15.23:43786) 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 -- 172.21.15.23:0/1154679008 learned_addr learned my addr 172.21.15.23:0/1154679008 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f575ffff640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 0x7f5768197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f576659c640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 0x7f5768197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 -- 172.21.15.23:0/1154679008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 msgr2=0x7f5768197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 0x7f5768197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 -- 172.21.15.23:0/1154679008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 msgr2=0x7f5768197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 0x7f5768197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 -- 172.21.15.23:0/1154679008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576819f570 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.961+0000 7f5766d9d640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5760037040 tx=0x7f5760032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.962+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760041020 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.962+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5760034d90 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.962+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576819f800 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.962+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5760039040 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.962+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576819fd40 con 0x7f5768102170 2024-03-28T15:50:46.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.963+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576819ffa0 con 0x7f5768102170 2024-03-28T15:50:46.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.964+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5760044020 con 0x7f5768102170 2024-03-28T15:50:46.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.964+0000 7f575f7fe640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 0x7f57540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:46.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.964+0000 7f576659c640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 0x7f57540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:46.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.964+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f57600f0b00 con 0x7f5768102170 2024-03-28T15:50:46.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.965+0000 7f576659c640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 0x7f57540a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5758005fd0 tx=0x7f575800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:46.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:46.969+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f576004b020 con 0x7f5768102170 2024-03-28T15:50:47.150 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.151+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57681a0170 con 0x7f5768102170 2024-03-28T15:50:47.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.151+0000 7f575f7fe640 1 -- 172.21.15.23:0/1154679008 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57600bd960 con 0x7f5768102170 2024-03-28T15:50:47.151 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:50:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 msgr2=0x7f57540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 0x7f57540a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5758005fd0 tx=0x7f575800b410 comp rx=0 tx=0).stop 2024-03-28T15:50:47.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 msgr2=0x7f576819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5760037040 tx=0x7f5760032f20 comp rx=0 tx=0).stop 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 shutdown_connections 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57540a2880 0x7f57540a4d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5768102170 0x7f576819ef00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57681017b0 0x7f5768197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 --2- 172.21.15.23:0/1154679008 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57681005c0 0x7f5768197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 >> 172.21.15.23:0/1154679008 conn(0x7f57680fbb50 msgr2=0x7f57680fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 shutdown_connections 2024-03-28T15:50:47.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.153+0000 7f576ce59640 1 -- 172.21.15.23:0/1154679008 wait complete. 2024-03-28T15:50:47.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:50:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- 172.21.15.23:0/2609579836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f81017b0 msgr2=0x7f90f8101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/2609579836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f81017b0 0x7f90f8101c30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f90f4009860 tx=0x7f90f402bb70 comp rx=0 tx=0).stop 2024-03-28T15:50:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- 172.21.15.23:0/2609579836 shutdown_connections 2024-03-28T15:50:47.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/2609579836 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f8102170 0x7f90f8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/2609579836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f81017b0 0x7f90f8101c30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/2609579836 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81005c0 0x7f90f81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- 172.21.15.23:0/2609579836 >> 172.21.15.23:0/2609579836 conn(0x7f90f80fbb50 msgr2=0x7f90f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- 172.21.15.23:0/2609579836 shutdown_connections 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- 172.21.15.23:0/2609579836 wait complete. 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 Processor -- start 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.270+0000 7f90ff8ac640 1 -- start start 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81017b0 0x7f90f8102b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f8102170 0x7f90f81030d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f810c220 con 0x7f90f81005c0 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f810c0a0 con 0x7f90f8102170 2024-03-28T15:50:47.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90f810c3a0 con 0x7f90f81017b0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43806/0 (socket says 172.21.15.23:43806) 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 -- 172.21.15.23:0/993819844 learned_addr learned my addr 172.21.15.23:0/993819844 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fde49640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f8102170 0x7f90f81030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fce47640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81017b0 0x7f90f8102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 -- 172.21.15.23:0/993819844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81017b0 msgr2=0x7f90f8102b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81017b0 0x7f90f8102b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 -- 172.21.15.23:0/993819844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f8102170 msgr2=0x7f90f81030d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f8102170 0x7f90f81030d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 -- 172.21.15.23:0/993819844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f8103960 con 0x7f90f81005c0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90fd648640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f90e800d950 tx=0x7f90e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e8017020 con 0x7f90f81005c0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f81a1af0 con 0x7f90f81005c0 2024-03-28T15:50:47.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90e8009c30 con 0x7f90f81005c0 2024-03-28T15:50:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e800f740 con 0x7f90f81005c0 2024-03-28T15:50:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.271+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90f81a2010 con 0x7f90f81005c0 2024-03-28T15:50:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.272+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90f81a21e0 con 0x7f90f81005c0 2024-03-28T15:50:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.273+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f90e801a020 con 0x7f90f81005c0 2024-03-28T15:50:47.275 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.274+0000 7f90ee7fc640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 0x7f90e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.274+0000 7f90fce47640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 0x7f90e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.274+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f90e80ce9d0 con 0x7f90f81005c0 2024-03-28T15:50:47.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.275+0000 7f90fce47640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 0x7f90e40a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f90f402c180 tx=0x7f90f4034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:47.278 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.278+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f90e8021070 con 0x7f90f81005c0 2024-03-28T15:50:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.463+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f90f81a23b0 con 0x7f90f81005c0 2024-03-28T15:50:47.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.463+0000 7f90ee7fc640 1 -- 172.21.15.23:0/993819844 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v90) v1 ==== 103+0+11 (secure 0 0 0) 0x7f90e8009de0 con 0x7f90f81005c0 2024-03-28T15:50:47.464 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 msgr2=0x7f90e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 0x7f90e40a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f90f402c180 tx=0x7f90f4034ad0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 msgr2=0x7f90f8104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f90e800d950 tx=0x7f90e800de10 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 shutdown_connections 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f90e40a2800 0x7f90e40a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90f8102170 0x7f90f81030d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f90f81017b0 0x7f90f8102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 --2- 172.21.15.23:0/993819844 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f90f81005c0 0x7f90f8104550 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.466 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 >> 172.21.15.23:0/993819844 conn(0x7f90f80fbb50 msgr2=0x7f90f80fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:47.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 shutdown_connections 2024-03-28T15:50:47.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.465+0000 7f90ff8ac640 1 -- 172.21.15.23:0/993819844 wait complete. 2024-03-28T15:50:47.477 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T15:50:47.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 -- 172.21.15.23:0/1717334039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff934102140 msgr2=0x7ff934104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/1717334039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff934102140 0x7ff934104530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff930009900 tx=0x7ff93002bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 -- 172.21.15.23:0/1717334039 shutdown_connections 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/1717334039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9341073a0 0x7ff9341097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/1717334039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934104a70 0x7ff934106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/1717334039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff934102140 0x7ff934104530 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 -- 172.21.15.23:0/1717334039 >> 172.21.15.23:0/1717334039 conn(0x7ff9340fbb50 msgr2=0x7ff9340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 -- 172.21.15.23:0/1717334039 shutdown_connections 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.580+0000 7ff93b5a9640 1 -- 172.21.15.23:0/1717334039 wait complete. 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 Processor -- start 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 -- start start 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934102140 0x7ff934197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff934104a70 0x7ff934197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410d360 con 0x7ff9341073a0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410d1e0 con 0x7ff934104a70 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff93b5a9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93410d4e0 con 0x7ff934102140 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43822/0 (socket says 172.21.15.23:43822) 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 -- 172.21.15.23:0/2886733642 learned_addr learned my addr 172.21.15.23:0/2886733642 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939345640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934102140 0x7ff934197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff938b44640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff934104a70 0x7ff934197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 -- 172.21.15.23:0/2886733642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934102140 msgr2=0x7ff934197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934102140 0x7ff934197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 -- 172.21.15.23:0/2886733642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff934104a70 msgr2=0x7ff934197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff934104a70 0x7ff934197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 -- 172.21.15.23:0/2886733642 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff93419f660 con 0x7ff9341073a0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.581+0000 7ff939b46640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00eb10 tx=0x7ff92c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:47.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.582+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c004b20 con 0x7ff9341073a0 2024-03-28T15:50:47.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.582+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff92c004cc0 con 0x7ff9341073a0 2024-03-28T15:50:47.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.582+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff93419f950 con 0x7ff9341073a0 2024-03-28T15:50:47.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.582+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c010760 con 0x7ff9341073a0 2024-03-28T15:50:47.585 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.582+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93419fea0 con 0x7ff9341073a0 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.583+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9340fd640 con 0x7ff9341073a0 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.584+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff92c005020 con 0x7ff9341073a0 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.585+0000 7ff92a7fc640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 0x7ff9200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.585+0000 7ff939345640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 0x7ff9200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.585+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff92c0cf360 con 0x7ff9341073a0 2024-03-28T15:50:47.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.585+0000 7ff939345640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 0x7ff9200a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff93002c7b0 tx=0x7ff930005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.589+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff92c09c1c0 con 0x7ff9341073a0 2024-03-28T15:50:47.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.785+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff93402d070 con 0x7ff9341073a0 2024-03-28T15:50:47.785 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.785+0000 7ff92a7fc640 1 -- 172.21.15.23:0/2886733642 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v90) v1 ==== 81+0+591 (secure 0 0 0) 0x7ff92c09bac0 con 0x7ff9341073a0 2024-03-28T15:50:47.785 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2505463471 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:47.785 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3395482026 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2095255310 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1534223304 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/611179097 2024-03-29T15:43:06.191436+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1777550602 2024-03-29T15:43:34.285406+0000 2024-03-28T15:50:47.786 INFO:teuthology.orchestra.run.smithi023.stderr:listed 10 entries 2024-03-28T15:50:47.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 msgr2=0x7ff9200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 0x7ff9200a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff93002c7b0 tx=0x7ff930005fd0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 msgr2=0x7ff93419eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00eb10 tx=0x7ff92c00c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 shutdown_connections 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff9200a2800 0x7ff9200a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9341073a0 0x7ff93419eff0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff934104a70 0x7ff934197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 --2- 172.21.15.23:0/2886733642 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff934102140 0x7ff934197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 >> 172.21.15.23:0/2886733642 conn(0x7ff9340fbb50 msgr2=0x7ff934102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 shutdown_connections 2024-03-28T15:50:47.788 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:47.787+0000 7ff93b5a9640 1 -- 172.21.15.23:0/2886733642 wait complete. 2024-03-28T15:50:47.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:47 smithi023 ceph-mon[32448]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 515 B/s wr, 0 op/s 2024-03-28T15:50:47.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2642183536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4189854976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1154679008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/993819844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.798 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi023.front.sepia.ceph.com... 2024-03-28T15:50:47.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:47.799 DEBUG:teuthology.orchestra.run.smithi023:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-28T15:50:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:47 smithi117 ceph-mon[36002]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 515 B/s wr, 0 op/s 2024-03-28T15:50:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2642183536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4189854976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1154679008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/993819844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi023.stderr:ceph-fuse[75645]: fuse finished with error 0 and tester_r 0 2024-03-28T15:50:47.825 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T15:50:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:47 smithi120 ceph-mon[36826]: pgmap v296: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 515 B/s wr, 0 op/s 2024-03-28T15:50:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2642183536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4189854976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1154679008' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:50:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/993819844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:50:48.746 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2886733642' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2886733642' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2886733642' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T15:50:49.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:49 smithi117 ceph-mon[36002]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-28T15:50:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:49 smithi120 ceph-mon[36826]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-28T15:50:49.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:49 smithi023 ceph-mon[32448]: pgmap v297: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 281 B/s wr, 0 op/s 2024-03-28T15:50:51.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:51 smithi117 ceph-mon[36002]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:51.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:51 smithi120 ceph-mon[36826]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:51.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:51 smithi023 ceph-mon[32448]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:53 smithi117 ceph-mon[36002]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:53 smithi120 ceph-mon[36826]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:53.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:53 smithi023 ceph-mon[32448]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:53.926 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:50:53.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:50:53.927 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-28T15:50:53.951 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:54.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 -- 172.21.15.23:0/450032024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1017b0 msgr2=0x7fdd6c101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/450032024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c101c30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009900 tx=0x7fdd5402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:54.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 -- 172.21.15.23:0/450032024 shutdown_connections 2024-03-28T15:50:54.109 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/450032024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c102170 0x7fdd6c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/450032024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c101c30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/450032024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 -- 172.21.15.23:0/450032024 >> 172.21.15.23:0/450032024 conn(0x7fdd6c0fbb30 msgr2=0x7fdd6c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 -- 172.21.15.23:0/450032024 shutdown_connections 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.105+0000 7fdd70bd4640 1 -- 172.21.15.23:0/450032024 wait complete. 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 Processor -- start 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 -- start start 2024-03-28T15:50:54.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c196df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c10c240 con 0x7fdd6c102170 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c10c0c0 con 0x7fdd6c1017b0 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd70bd4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c10c3c0 con 0x7fdd6c1005c0 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.106+0000 7fdd6ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33628/0 (socket says 172.21.15.23:33628) 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd61d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c196df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 -- 172.21.15.23:0/3730173957 learned_addr learned my addr 172.21.15.23:0/3730173957 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6a59c640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 -- 172.21.15.23:0/3730173957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 msgr2=0x7fdd6c1968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 -- 172.21.15.23:0/3730173957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 msgr2=0x7fdd6c196df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c196df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 -- 172.21.15.23:0/3730173957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c19e4d0 con 0x7fdd6c102170 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6a59c640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd61d9b640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd6ad9d640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400eb10 tx=0x7fdd6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64004b20 con 0x7fdd6c102170 2024-03-28T15:50:54.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd64004cc0 con 0x7fdd6c102170 2024-03-28T15:50:54.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c19e7c0 con 0x7fdd6c102170 2024-03-28T15:50:54.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd64010760 con 0x7fdd6c102170 2024-03-28T15:50:54.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.107+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c19ec90 con 0x7fdd6c102170 2024-03-28T15:50:54.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.108+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c1078e0 con 0x7fdd6c102170 2024-03-28T15:50:54.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.109+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdd64005020 con 0x7fdd6c102170 2024-03-28T15:50:54.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.110+0000 7fdd63fff640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 0x7fdd580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.110+0000 7fdd6a59c640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 0x7fdd580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.110+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdd640cf220 con 0x7fdd6c102170 2024-03-28T15:50:54.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.111+0000 7fdd6a59c640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 0x7fdd580a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c101610 tx=0x7fdd5c005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.115+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd6409c120 con 0x7fdd6c102170 2024-03-28T15:50:54.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.299+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd6c0622a0 con 0x7fdd6c102170 2024-03-28T15:50:54.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.300+0000 7fdd63fff640 1 -- 172.21.15.23:0/3730173957 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7fdd6409ba20 con 0x7fdd6c102170 2024-03-28T15:50:54.301 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:54.301 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 msgr2=0x7fdd580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 0x7fdd580a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c101610 tx=0x7fdd5c005e70 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 msgr2=0x7fdd6c19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400eb10 tx=0x7fdd6400c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 shutdown_connections 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fdd580a2800 0x7fdd580a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd6c102170 0x7fdd6c19de60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd6c1017b0 0x7fdd6c196df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 --2- 172.21.15.23:0/3730173957 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd6c1005c0 0x7fdd6c1968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 >> 172.21.15.23:0/3730173957 conn(0x7fdd6c0fbb30 msgr2=0x7fdd6c104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 shutdown_connections 2024-03-28T15:50:54.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.302+0000 7fdd70bd4640 1 -- 172.21.15.23:0/3730173957 wait complete. 2024-03-28T15:50:54.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 -- 172.21.15.23:0/3580521452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb158101830 msgr2=0x7fb158101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 --2- 172.21.15.23:0/3580521452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb158101830 0x7fb158101cb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fb148009900 tx=0x7fb14802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 -- 172.21.15.23:0/3580521452 shutdown_connections 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 --2- 172.21.15.23:0/3580521452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1581021f0 0x7fb158108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 --2- 172.21.15.23:0/3580521452 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb158101830 0x7fb158101cb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 --2- 172.21.15.23:0/3580521452 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158100640 0x7fb158100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 -- 172.21.15.23:0/3580521452 >> 172.21.15.23:0/3580521452 conn(0x7fb1580fbbd0 msgr2=0x7fb1580fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 -- 172.21.15.23:0/3580521452 shutdown_connections 2024-03-28T15:50:54.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.416+0000 7fb15e709640 1 -- 172.21.15.23:0/3580521452 wait complete. 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 Processor -- start 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 -- start start 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 0x7fb158196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158101830 0x7fb158196e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15810c250 con 0x7fb1581021f0 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15810c0d0 con 0x7fb158100640 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15e709640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15810c3d0 con 0x7fb158101830 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33636/0 (socket says 172.21.15.23:33636) 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 -- 172.21.15.23:0/2261363967 learned_addr learned my addr 172.21.15.23:0/2261363967 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb157fff640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 0x7fb158196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb1577fe640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158101830 0x7fb158196e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.420 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 -- 172.21.15.23:0/2261363967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158101830 msgr2=0x7fb158196e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158101830 0x7fb158196e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 -- 172.21.15.23:0/2261363967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 msgr2=0x7fb158196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 0x7fb158196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb15cca6640 1 -- 172.21.15.23:0/2261363967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15819e530 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.417+0000 7fb157fff640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 0x7fb158196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb15cca6640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb15000efd0 tx=0x7fb15000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150003b70 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15819e820 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb150003d10 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500109f0 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.418+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15819ecf0 con 0x7fb1581021f0 2024-03-28T15:50:54.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.419+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb158107a30 con 0x7fb1581021f0 2024-03-28T15:50:54.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.420+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb150004020 con 0x7fb1581021f0 2024-03-28T15:50:54.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.420+0000 7fb1557fa640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 0x7fb1400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.421+0000 7fb157fff640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 0x7fb1400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.421+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb1500cf1c0 con 0x7fb1581021f0 2024-03-28T15:50:54.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.421+0000 7fb157fff640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 0x7fb1400a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0058c0 tx=0x7fb14c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.425+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb15009c020 con 0x7fb1581021f0 2024-03-28T15:50:54.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.609+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1580622a0 con 0x7fb1581021f0 2024-03-28T15:50:54.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.611+0000 7fb1557fa640 1 -- 172.21.15.23:0/2261363967 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7fb15009b920 con 0x7fb1581021f0 2024-03-28T15:50:54.611 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:50:54.611 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 msgr2=0x7fb1400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 0x7fb1400a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb14c0058c0 tx=0x7fb14c00a2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 msgr2=0x7fb15819dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb15000efd0 tx=0x7fb15000c6a0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 shutdown_connections 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb1400a2800 0x7fb1400a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb1581021f0 0x7fb15819dec0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.613 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb158101830 0x7fb158196e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 --2- 172.21.15.23:0/2261363967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb158100640 0x7fb158196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 >> 172.21.15.23:0/2261363967 conn(0x7fb1580fbbd0 msgr2=0x7fb158104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:54.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 shutdown_connections 2024-03-28T15:50:54.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.613+0000 7fb15e709640 1 -- 172.21.15.23:0/2261363967 wait complete. 2024-03-28T15:50:54.623 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3730173957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:54.624 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-28T15:50:54.732 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 -- 172.21.15.23:0/3750019270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 msgr2=0x7f1bfc1016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3750019270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1016b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4009ca0 tx=0x7f1bf402c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 -- 172.21.15.23:0/3750019270 shutdown_connections 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3750019270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1016b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3750019270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc100890 0x7f1bfc100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3750019270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc0fff30 0x7f1bfc100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 -- 172.21.15.23:0/3750019270 >> 172.21.15.23:0/3750019270 conn(0x7f1bfc0fbb30 msgr2=0x7f1bfc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 -- 172.21.15.23:0/3750019270 shutdown_connections 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.730+0000 7f1c038ea640 1 -- 172.21.15.23:0/3750019270 wait complete. 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 Processor -- start 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 -- start start 2024-03-28T15:50:54.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc0fff30 0x7f1bfc199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc100890 0x7f1bfc19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc10f580 con 0x7f1bfc101230 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc10f400 con 0x7f1bfc100890 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c038ea640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bfc10f700 con 0x7f1bfc0fff30 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33662/0 (socket says 172.21.15.23:33662) 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 -- 172.21.15.23:0/3080081586 learned_addr learned my addr 172.21.15.23:0/3080081586 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c00e85640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc100890 0x7f1bfc19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01686640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc0fff30 0x7f1bfc199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 -- 172.21.15.23:0/3080081586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc0fff30 msgr2=0x7f1bfc199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc0fff30 0x7f1bfc199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 -- 172.21.15.23:0/3080081586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc100890 msgr2=0x7f1bfc19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:54.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc100890 0x7f1bfc19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 -- 172.21.15.23:0/3080081586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bfc1a17c0 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1c01e87640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4037040 tx=0x7f1bf4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.731+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4041020 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.732+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1bf4034cd0 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.732+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf4039040 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.732+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc1a1a50 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.732+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bfc1a1ef0 con 0x7f1bfc101230 2024-03-28T15:50:54.735 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.733+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bfc10add0 con 0x7f1bfc101230 2024-03-28T15:50:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.734+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1bf4044020 con 0x7f1bfc101230 2024-03-28T15:50:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.734+0000 7f1bf27fc640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 0x7f1be80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.734+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1bf40f1040 con 0x7f1bfc101230 2024-03-28T15:50:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.734+0000 7f1c01686640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 0x7f1be80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:54.736 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.735+0000 7f1c01686640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 0x7f1be80a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009b60 tx=0x7f1bf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:54.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.738+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1bf404b020 con 0x7f1bfc101230 2024-03-28T15:50:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:54 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3730173957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:54 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3730173957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:54.958 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:54.958+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1bfc02d070 con 0x7f1bfc101230 2024-03-28T15:50:55.526 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.525+0000 7f1bf27fc640 1 -- 172.21.15.23:0/3080081586 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1bf404e350 con 0x7f1bfc101230 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 msgr2=0x7f1be80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 0x7f1be80a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009b60 tx=0x7f1bf8005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 msgr2=0x7f1bfc1a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1bf4037040 tx=0x7f1bf4032f20 comp rx=0 tx=0).stop 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 shutdown_connections 2024-03-28T15:50:55.528 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1be80a2800 0x7f1be80a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1bfc101230 0x7f1bfc1a1150 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bfc100890 0x7f1bfc19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 --2- 172.21.15.23:0/3080081586 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1bfc0fff30 0x7f1bfc199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 >> 172.21.15.23:0/3080081586 conn(0x7f1bfc0fbb30 msgr2=0x7f1bfc0fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 shutdown_connections 2024-03-28T15:50:55.529 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.528+0000 7f1c038ea640 1 -- 172.21.15.23:0/3080081586 wait complete. 2024-03-28T15:50:55.540 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-28T15:50:55.541 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-28T15:50:55.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 -- 172.21.15.23:0/2830417658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a01033f0 msgr2=0x7fe8a0109c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/2830417658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a01033f0 0x7fe8a0109c80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe898009cd0 tx=0x7fe89802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.651 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 -- 172.21.15.23:0/2830417658 shutdown_connections 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/2830417658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a01033f0 0x7fe8a0109c80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/2830417658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a0102a30 0x7fe8a0102eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/2830417658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a00ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 -- 172.21.15.23:0/2830417658 >> 172.21.15.23:0/2830417658 conn(0x7fe8a00fb2b0 msgr2=0x7fe8a00fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 -- 172.21.15.23:0/2830417658 shutdown_connections 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.649+0000 7fe8a5c81640 1 -- 172.21.15.23:0/2830417658 wait complete. 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 Processor -- start 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 -- start start 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a0197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 0x7fe8a019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a010d340 con 0x7fe8a0102a30 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a010d1c0 con 0x7fe8a00ff9c0 2024-03-28T15:50:55.652 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe8a5c81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a010d4c0 con 0x7fe8a01033f0 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33668/0 (socket says 172.21.15.23:33668) 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 -- 172.21.15.23:0/4001939169 learned_addr learned my addr 172.21.15.23:0/4001939169 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89f7fe640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a0197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89ffff640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 0x7fe8a019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 -- 172.21.15.23:0/4001939169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 msgr2=0x7fe8a019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 0x7fe8a019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 -- 172.21.15.23:0/4001939169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 msgr2=0x7fe8a0197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a0197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 -- 172.21.15.23:0/4001939169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a019f570 con 0x7fe8a0102a30 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89f7fe640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a0197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89ffff640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 0x7fe8a019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.650+0000 7fe89effd640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe89400e9f0 tx=0x7fe89400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.651+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400cd80 con 0x7fe8a0102a30 2024-03-28T15:50:55.653 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.651+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe894010910 con 0x7fe8a0102a30 2024-03-28T15:50:55.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.651+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a019f860 con 0x7fe8a0102a30 2024-03-28T15:50:55.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.651+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894018990 con 0x7fe8a0102a30 2024-03-28T15:50:55.654 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.651+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a019fdb0 con 0x7fe8a0102a30 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.652+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a0108b00 con 0x7fe8a0102a30 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.653+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe894010ac0 con 0x7fe8a0102a30 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.653+0000 7fe89cff9640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 0x7fe8880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.654+0000 7fe89f7fe640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 0x7fe8880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.654+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe8940cf0a0 con 0x7fe8a0102a30 2024-03-28T15:50:55.655 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.654+0000 7fe89f7fe640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 0x7fe8880a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0069a10 tx=0x7fe890005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:55.658 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.658+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe89409bf00 con 0x7fe8a0102a30 2024-03-28T15:50:55.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:55 smithi117 ceph-mon[36002]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:55.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:55 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2261363967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:55.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:55 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T15:50:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:55 smithi120 ceph-mon[36826]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:55 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2261363967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:55 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T15:50:55.881 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:55.881+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7fe8a00622d0 con 0x7fe8a0102a30 2024-03-28T15:50:55.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:55 smithi023 ceph-mon[32448]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:55.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:55 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2261363967' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:50:55.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:55 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-28T15:50:56.541 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.541+0000 7fe89cff9640 1 -- 172.21.15.23:0/4001939169 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7fe89409b800 con 0x7fe8a0102a30 2024-03-28T15:50:56.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 msgr2=0x7fe8880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:56.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 0x7fe8880a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe8a0069a10 tx=0x7fe890005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 msgr2=0x7fe8a0197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe89400e9f0 tx=0x7fe89400eeb0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 shutdown_connections 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8880a2800 0x7fe8880a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe8a01033f0 0x7fe8a019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe8a0102a30 0x7fe8a0197e90 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 --2- 172.21.15.23:0/4001939169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe8a00ff9c0 0x7fe8a0197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 >> 172.21.15.23:0/4001939169 conn(0x7fe8a00fb2b0 msgr2=0x7fe8a0105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 shutdown_connections 2024-03-28T15:50:56.544 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.543+0000 7fe8a5c81640 1 -- 172.21.15.23:0/4001939169 wait complete. 2024-03-28T15:50:56.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-28T15:50:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- 172.21.15.23:0/1053025048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 msgr2=0x7fcb9c101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 --2- 172.21.15.23:0/1053025048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c101ac0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009900 tx=0x7fcb9802bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- 172.21.15.23:0/1053025048 shutdown_connections 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 --2- 172.21.15.23:0/1053025048 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c109ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 --2- 172.21.15.23:0/1053025048 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c102890 0x7fcb9c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 --2- 172.21.15.23:0/1053025048 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c101ac0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- 172.21.15.23:0/1053025048 >> 172.21.15.23:0/1053025048 conn(0x7fcb9c0fcc10 msgr2=0x7fcb9c0ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- 172.21.15.23:0/1053025048 shutdown_connections 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- 172.21.15.23:0/1053025048 wait complete. 2024-03-28T15:50:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 Processor -- start 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.667+0000 7fcba34dc640 1 -- start start 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c102890 0x7fcb9c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c10d2f0 con 0x7fcb9c1016a0 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c10d170 con 0x7fcb9c103250 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c10d470 con 0x7fcb9c102890 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33694/0 (socket says 172.21.15.23:33694) 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 -- 172.21.15.23:0/4118605219 learned_addr learned my addr 172.21.15.23:0/4118605219 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1a79640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 -- 172.21.15.23:0/4118605219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c102890 msgr2=0x7fcb9c073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c102890 0x7fcb9c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 -- 172.21.15.23:0/4118605219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 msgr2=0x7fcb9c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 -- 172.21.15.23:0/4118605219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c074860 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1a79640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba1278640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb98005ab0 tx=0x7fcb98005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98041070 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb980054e0 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c1a1a20 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98034440 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.668+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c1a1f70 con 0x7fcb9c1016a0 2024-03-28T15:50:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.669+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c108a40 con 0x7fcb9c1016a0 2024-03-28T15:50:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.670+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcb98044070 con 0x7fcb9c1016a0 2024-03-28T15:50:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.671+0000 7fcb927fc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 0x7fcb880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:56.673 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.671+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcb980f09a0 con 0x7fcb9c1016a0 2024-03-28T15:50:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.671+0000 7fcba0a77640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 0x7fcb880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:56.674 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.672+0000 7fcba0a77640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 0x7fcb880a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0751c0 tx=0x7fcb8c005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:56.676 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.676+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb980bd800 con 0x7fcb9c1016a0 2024-03-28T15:50:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:56 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T15:50:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:56 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:56 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T15:50:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:56 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T15:50:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:56 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:56 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T15:50:56.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:56.898+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fcb9c02d290 con 0x7fcb9c1016a0 2024-03-28T15:50:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:56 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3080081586' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-28T15:50:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:56 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:56 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-28T15:50:57.552 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.552+0000 7fcb927fc640 1 -- 172.21.15.23:0/4118605219 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7fcb980bd100 con 0x7fcb9c1016a0 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.554+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 msgr2=0x7fcb880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.554+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 0x7fcb880a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0751c0 tx=0x7fcb8c005c70 comp rx=0 tx=0).stop 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 msgr2=0x7fcb9c075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcb98005ab0 tx=0x7fcb98005dc0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 shutdown_connections 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb880a2800 0x7fcb880a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb9c103250 0x7fcb9c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.555 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb9c102890 0x7fcb9c073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 --2- 172.21.15.23:0/4118605219 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb9c1016a0 0x7fcb9c075450 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 >> 172.21.15.23:0/4118605219 conn(0x7fcb9c0fcc10 msgr2=0x7fcb9c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:57.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 shutdown_connections 2024-03-28T15:50:57.556 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.555+0000 7fcba34dc640 1 -- 172.21.15.23:0/4118605219 wait complete. 2024-03-28T15:50:57.571 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-28T15:50:57.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- 172.21.15.23:0/3227342260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b90102160 msgr2=0x7f8b901089f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 --2- 172.21.15.23:0/3227342260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b90102160 0x7f8b901089f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8b88009ca0 tx=0x7f8b8802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- 172.21.15.23:0/3227342260 shutdown_connections 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 --2- 172.21.15.23:0/3227342260 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b90102160 0x7f8b901089f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 --2- 172.21.15.23:0/3227342260 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b901017a0 0x7f8b90101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 --2- 172.21.15.23:0/3227342260 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b901009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- 172.21.15.23:0/3227342260 >> 172.21.15.23:0/3227342260 conn(0x7f8b900fbb40 msgr2=0x7f8b900fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- 172.21.15.23:0/3227342260 shutdown_connections 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- 172.21.15.23:0/3227342260 wait complete. 2024-03-28T15:50:57.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 Processor -- start 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.680+0000 7f8b94b76640 1 -- start start 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b90197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b90102160 0x7f8b9019ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9010c210 con 0x7f8b901017a0 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9010c090 con 0x7f8b90102160 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b9010c390 con 0x7f8b901005b0 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:33700/0 (socket says 172.21.15.23:33700) 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 -- 172.21.15.23:0/4057215423 learned_addr learned my addr 172.21.15.23:0/4057215423 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b8e59c640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b90197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b8ed9d640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b90102160 0x7f8b9019ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 -- 172.21.15.23:0/4057215423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 msgr2=0x7f8b90197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b90197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 -- 172.21.15.23:0/4057215423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b90102160 msgr2=0x7f8b9019ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b90102160 0x7f8b9019ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 -- 172.21.15.23:0/4057215423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9019f590 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b8e59c640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b90197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b87fff640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800ea60 tx=0x7f8b7800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7800cde0 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b780035f0 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b9019f880 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78010670 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.681+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b9019fdd0 con 0x7f8b901017a0 2024-03-28T15:50:57.685 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.682+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b901a0030 con 0x7f8b901017a0 2024-03-28T15:50:57.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.683+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8b780037a0 con 0x7f8b901017a0 2024-03-28T15:50:57.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.684+0000 7f8b877fe640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 0x7f8b7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:57.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.684+0000 7f8b8e59c640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 0x7f8b7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:57.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.684+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8b780cf0d0 con 0x7f8b901017a0 2024-03-28T15:50:57.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.684+0000 7f8b8e59c640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 0x7f8b7c0a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8b90101600 tx=0x7f8b80005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:57.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.688+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8b7809bf30 con 0x7f8b901017a0 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T15:50:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:57 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T15:50:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:57 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T15:50:57.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:57.911+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f8b901a0420 con 0x7f8b901017a0 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: pgmap v301: 129 pgs: 129 active+clean; 581 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4001939169' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-28T15:50:57.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:57 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-28T15:50:58.554 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T15:50:58.255+0000 7f6e41e4c700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-28T15:50:58.562 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.562+0000 7f8b877fe640 1 -- 172.21.15.23:0/4057215423 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8b7809b830 con 0x7f8b901017a0 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 msgr2=0x7f8b7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 0x7f8b7c0a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8b90101600 tx=0x7f8b80005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 msgr2=0x7f8b90197eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800ea60 tx=0x7f8b7800ef20 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 shutdown_connections 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8b7c0a2800 0x7f8b7c0a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b90102160 0x7f8b9019ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8b901017a0 0x7f8b90197eb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 --2- 172.21.15.23:0/4057215423 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8b901005b0 0x7f8b90197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.565 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.564+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 >> 172.21.15.23:0/4057215423 conn(0x7f8b900fbb40 msgr2=0x7f8b900fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.565+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 shutdown_connections 2024-03-28T15:50:58.566 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.565+0000 7f8b94b76640 1 -- 172.21.15.23:0/4057215423 wait complete. 2024-03-28T15:50:58.582 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-28T15:50:58.583 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.691+0000 7fd65c137640 1 -- 172.21.15.23:0/1432386181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd654102150 msgr2=0x7fd6541089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.691+0000 7fd65c137640 1 --2- 172.21.15.23:0/1432386181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd654102150 0x7fd6541089e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd64c009ca0 tx=0x7fd64c02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.691+0000 7fd65c137640 1 -- 172.21.15.23:0/1432386181 shutdown_connections 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- 172.21.15.23:0/1432386181 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd654102150 0x7fd6541089e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- 172.21.15.23:0/1432386181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654101790 0x7fd654101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- 172.21.15.23:0/1432386181 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd6541005a0 0x7fd6541009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- 172.21.15.23:0/1432386181 >> 172.21.15.23:0/1432386181 conn(0x7fd6540fbb30 msgr2=0x7fd6540fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- 172.21.15.23:0/1432386181 shutdown_connections 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- 172.21.15.23:0/1432386181 wait complete. 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 Processor -- start 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- start start 2024-03-28T15:50:58.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654101790 0x7fd654197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654102150 0x7fd65419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410c230 con 0x7fd6541005a0 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410c0b0 con 0x7fd654102150 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd65c137640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd65410c3b0 con 0x7fd654101790 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd659ed3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd659ed3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40190/0 (socket says 172.21.15.23:40190) 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd659ed3640 1 -- 172.21.15.23:0/2925160437 learned_addr learned my addr 172.21.15.23:0/2925160437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.692+0000 7fd6596d2640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654101790 0x7fd654197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd65a6d4640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654102150 0x7fd65419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 -- 172.21.15.23:0/2925160437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654101790 msgr2=0x7fd654197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:58.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654101790 0x7fd654197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 -- 172.21.15.23:0/2925160437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654102150 msgr2=0x7fd65419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654102150 0x7fd65419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 -- 172.21.15.23:0/2925160437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65419f5c0 con 0x7fd6541005a0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd659ed3640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd65000e9c0 tx=0x7fd65000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65000cd80 con 0x7fd6541005a0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6500035f0 con 0x7fd6541005a0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd65419f8b0 con 0x7fd6541005a0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650010690 con 0x7fd6541005a0 2024-03-28T15:50:58.696 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.693+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65419fdb0 con 0x7fd6541005a0 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.694+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6541a0010 con 0x7fd6541005a0 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.695+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd6500037a0 con 0x7fd6541005a0 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.696+0000 7fd64affd640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 0x7fd6400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.696+0000 7fd6596d2640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 0x7fd6400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.696+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd6500cf000 con 0x7fd6541005a0 2024-03-28T15:50:58.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.696+0000 7fd6596d2640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 0x7fd6400a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6440095f0 tx=0x7fd644005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:58.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.700+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd65009be60 con 0x7fd6541005a0 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:active 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-28T15:50:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:58 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:active 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-28T15:50:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:58 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T15:50:58.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:58.909+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6541a01e0 con 0x7fd6400a2800 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4118605219' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] up:active 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-28T15:50:58.915 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-28T15:50:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:50:58.913+0000 7f280fb08700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:50:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:50:58.913+0000 7f280fb08700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:50:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:50:58.914+0000 7f280fb08700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:50:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:50:58.914+0000 7f280fb08700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:50:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:50:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:50:58.914+0000 7f280fb08700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:50:59.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.238+0000 7fd64affd640 1 -- 172.21.15.23:0/2925160437 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd6541a01e0 con 0x7fd6400a2800 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 msgr2=0x7fd6400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 0x7fd6400a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd6440095f0 tx=0x7fd644005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 msgr2=0x7fd6541979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fd65000e9c0 tx=0x7fd65000ee80 comp rx=0 tx=0).stop 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 shutdown_connections 2024-03-28T15:50:59.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd6400a2800 0x7fd6400a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd654102150 0x7fd65419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd654101790 0x7fd654197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 --2- 172.21.15.23:0/2925160437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd6541005a0 0x7fd6541979a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 >> 172.21.15.23:0/2925160437 conn(0x7fd6540fbb30 msgr2=0x7fd6540fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 shutdown_connections 2024-03-28T15:50:59.242 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.241+0000 7fd65c137640 1 -- 172.21.15.23:0/2925160437 wait complete. 2024-03-28T15:50:59.252 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-28T15:50:59.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 -- 172.21.15.23:0/1242289757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc101790 msgr2=0x7f46cc101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 --2- 172.21.15.23:0/1242289757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc101790 0x7f46cc101c10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f46b4009900 tx=0x7f46b402bc30 comp rx=0 tx=0).stop 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 -- 172.21.15.23:0/1242289757 shutdown_connections 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 --2- 172.21.15.23:0/1242289757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc102150 0x7f46cc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 --2- 172.21.15.23:0/1242289757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc101790 0x7f46cc101c10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 --2- 172.21.15.23:0/1242289757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc1005a0 0x7f46cc1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 -- 172.21.15.23:0/1242289757 >> 172.21.15.23:0/1242289757 conn(0x7f46cc0fbb30 msgr2=0x7f46cc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.360+0000 7f46d092f640 1 -- 172.21.15.23:0/1242289757 shutdown_connections 2024-03-28T15:50:59.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 -- 172.21.15.23:0/1242289757 wait complete. 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 Processor -- start 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 -- start start 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 0x7f46cc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 0x7f46cc19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10c230 con 0x7f46cc102150 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10c0b0 con 0x7f46cc1005a0 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.361+0000 7f46d092f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc10c3b0 con 0x7f46cc101790 2024-03-28T15:50:59.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40224/0 (socket says 172.21.15.23:40224) 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 -- 172.21.15.23:0/1211057521 learned_addr learned my addr 172.21.15.23:0/1211057521 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46ca59c640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 0x7f46cc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46c3fff640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 0x7f46cc19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 -- 172.21.15.23:0/1211057521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 msgr2=0x7f46cc19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 0x7f46cc19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 -- 172.21.15.23:0/1211057521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 msgr2=0x7f46cc199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 0x7f46cc199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 -- 172.21.15.23:0/1211057521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc1a1830 con 0x7f46cc102150 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46c3fff640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 0x7f46cc19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46ca59c640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 0x7f46cc199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46cad9d640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f46c400eb10 tx=0x7f46c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4004b20 con 0x7f46cc102150 2024-03-28T15:50:59.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46c4004cc0 con 0x7f46cc102150 2024-03-28T15:50:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc1a1b20 con 0x7f46cc102150 2024-03-28T15:50:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4010760 con 0x7f46cc102150 2024-03-28T15:50:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.362+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc1a2020 con 0x7f46cc102150 2024-03-28T15:50:59.368 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.363+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc1a21f0 con 0x7f46cc102150 2024-03-28T15:50:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.364+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f46c4010900 con 0x7f46cc102150 2024-03-28T15:50:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.365+0000 7f46c37fe640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 0x7f46b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.365+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f46c40cf4d0 con 0x7f46cc102150 2024-03-28T15:50:59.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.365+0000 7f46ca59c640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 0x7f46b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.365+0000 7f46ca59c640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 0x7f46b80a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f46cc19b0f0 tx=0x7f46bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:59.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.369+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46c4053740 con 0x7f46cc102150 2024-03-28T15:50:59.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.583+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f46cc068f10 con 0x7f46b80a2800 2024-03-28T15:50:59.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.605+0000 7f46c37fe640 1 -- 172.21.15.23:0/1211057521 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46cc068f10 con 0x7f46b80a2800 2024-03-28T15:50:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 msgr2=0x7f46b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 0x7f46b80a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f46cc19b0f0 tx=0x7f46bc005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:59.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 msgr2=0x7f46cc1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f46c400eb10 tx=0x7f46c400c2d0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 shutdown_connections 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f46b80a2800 0x7f46b80a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f46cc102150 0x7f46cc1a1190 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f46cc101790 0x7f46cc19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 --2- 172.21.15.23:0/1211057521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f46cc1005a0 0x7f46cc199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 >> 172.21.15.23:0/1211057521 conn(0x7f46cc0fbb30 msgr2=0x7f46cc068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 shutdown_connections 2024-03-28T15:50:59.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.607+0000 7f46d092f640 1 -- 172.21.15.23:0/1211057521 wait complete. 2024-03-28T15:50:59.618 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-28T15:50:59.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.725+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/3127839050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 msgr2=0x7f1a08101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.728 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.725+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/3127839050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a08101710 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009be0 tx=0x7f19f802bf80 comp rx=0 tx=0).stop 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/3127839050 shutdown_connections 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/3127839050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a08101710 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/3127839050 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a08100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/3127839050 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 0x7f1a081003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/3127839050 >> 172.21.15.23:0/3127839050 conn(0x7f1a080fbb30 msgr2=0x7f1a080fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/3127839050 shutdown_connections 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/3127839050 wait complete. 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 Processor -- start 2024-03-28T15:50:59.729 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 -- start start 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.726+0000 7f1a0ca0b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 0x7f1a08199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a081a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0810f500 con 0x7f1a081008f0 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0810f380 con 0x7f1a08101290 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0810f680 con 0x7f1a080fff90 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40252/0 (socket says 172.21.15.23:40252) 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 -- 172.21.15.23:0/2420289998 learned_addr learned my addr 172.21.15.23:0/2420289998 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0659c640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 0x7f1a08199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.730 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a06d9d640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a081a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 -- 172.21.15.23:0/2420289998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 msgr2=0x7f1a08199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 0x7f1a08199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 -- 172.21.15.23:0/2420289998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 msgr2=0x7f1a081a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a081a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 -- 172.21.15.23:0/2420289998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a081a17a0 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a05d9b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19f000e9c0 tx=0x7f19f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f000cde0 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a081a1a90 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19f00035f0 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f0010650 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.727+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a081a1fe0 con 0x7f1a081008f0 2024-03-28T15:50:59.731 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.728+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a0810ae70 con 0x7f1a081008f0 2024-03-28T15:50:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.729+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f19f00037a0 con 0x7f1a081008f0 2024-03-28T15:50:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.730+0000 7f19ff7fe640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 0x7f19f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:50:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.730+0000 7f1a0659c640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 0x7f19f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:50:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.730+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f19f00cefa0 con 0x7f1a081008f0 2024-03-28T15:50:59.733 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.730+0000 7f1a0659c640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 0x7f19f40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009bd0 tx=0x7f1a00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:50:59.734 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.734+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19f009be00 con 0x7f1a081008f0 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:active 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-28T15:50:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:active 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: Cluster is now healthy 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T15:50:59.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:50:59 smithi117 ceph-mon[36002]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T15:50:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-28T15:50:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T15:50:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:active 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:50:59.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:active 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: Cluster is now healthy 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T15:50:59.841 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:50:59 smithi120 ceph-mon[36826]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 3 op/s 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4057215423' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] up:active 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2024-03-28T15:50:59.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] up:active 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: Cluster is now healthy 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 4 up:standby-replay 4 up:standby 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-28T15:50:59.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:50:59 smithi023 ceph-mon[32448]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-28T15:50:59.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.947+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a08061ae0 con 0x7f19f40a2800 2024-03-28T15:50:59.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.952+0000 7f19ff7fe640 1 -- 172.21.15.23:0/2420289998 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f1a08061ae0 con 0x7f19f40a2800 2024-03-28T15:50:59.953 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_0/7a5081f8-8171-4f53-9c4a-5b7443027335 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 msgr2=0x7f19f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 0x7f19f40a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a00009bd0 tx=0x7f1a00005f90 comp rx=0 tx=0).stop 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 msgr2=0x7f1a0819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f19f000e9c0 tx=0x7f19f000ee80 comp rx=0 tx=0).stop 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 shutdown_connections 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f19f40a2800 0x7f19f40a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a08101290 0x7f1a081a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1a081008f0 0x7f1a0819a0c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 --2- 172.21.15.23:0/2420289998 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1a080fff90 0x7f1a08199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:50:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 >> 172.21.15.23:0/2420289998 conn(0x7f1a080fbb30 msgr2=0x7f1a080fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:50:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 shutdown_connections 2024-03-28T15:50:59.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:50:59.955+0000 7f1a0ca0b640 1 -- 172.21.15.23:0/2420289998 wait complete. 2024-03-28T15:50:59.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-28T15:51:00.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 -- 172.21.15.23:0/3472044199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 msgr2=0x7f12e4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3472044199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e4102d30 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12cc009900 tx=0x7f12cc02bc30 comp rx=0 tx=0).stop 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 -- 172.21.15.23:0/3472044199 shutdown_connections 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3472044199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 0x7f12e4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3472044199 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e4102d30 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3472044199 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 -- 172.21.15.23:0/3472044199 >> 172.21.15.23:0/3472044199 conn(0x7f12e40fcc30 msgr2=0x7f12e40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 -- 172.21.15.23:0/3472044199 shutdown_connections 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.074+0000 7f12e88dd640 1 -- 172.21.15.23:0/3472044199 wait complete. 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 Processor -- start 2024-03-28T15:51:00.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 -- start start 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 0x7f12e41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410d390 con 0x7f12e41028b0 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410d210 con 0x7f12e4103270 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e88dd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e410d510 con 0x7f12e41016c0 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40266/0 (socket says 172.21.15.23:40266) 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 -- 172.21.15.23:0/3373924255 learned_addr learned my addr 172.21.15.23:0/3373924255 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e259c640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e2d9d640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 0x7f12e41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 -- 172.21.15.23:0/3373924255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 msgr2=0x7f12e4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 -- 172.21.15.23:0/3373924255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 msgr2=0x7f12e41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 0x7f12e41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12dbfff640 1 -- 172.21.15.23:0/3373924255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e41a1830 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.075+0000 7f12e259c640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12dbfff640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12cc000c00 tx=0x7f12cc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc043020 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e41a1ac0 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12cc0054e0 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc003c40 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.076+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e41a2010 con 0x7f12e41028b0 2024-03-28T15:51:00.080 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.077+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e41a21e0 con 0x7f12e41028b0 2024-03-28T15:51:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.078+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12cc046020 con 0x7f12e41028b0 2024-03-28T15:51:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.079+0000 7f12db7fe640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 0x7f12d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.079+0000 7f12e259c640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 0x7f12d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.079+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f12cc0f1190 con 0x7f12e41028b0 2024-03-28T15:51:00.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.079+0000 7f12e259c640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 0x7f12d00a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f12e419b0f0 tx=0x7f12d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.082 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.082+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12cc04d020 con 0x7f12e41028b0 2024-03-28T15:51:00.308 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.308+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f12e4074ff0 con 0x7f12d00a2800 2024-03-28T15:51:00.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.330+0000 7f12db7fe640 1 -- 172.21.15.23:0/3373924255 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f12e4074ff0 con 0x7f12d00a2800 2024-03-28T15:51:00.332 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 msgr2=0x7f12d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 0x7f12d00a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f12e419b0f0 tx=0x7f12d4005f90 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 msgr2=0x7f12e419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12cc000c00 tx=0x7f12cc02c650 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 shutdown_connections 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f12d00a2800 0x7f12d00a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12e4103270 0x7f12e41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f12e41028b0 0x7f12e419a160 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 --2- 172.21.15.23:0/3373924255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f12e41016c0 0x7f12e4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 >> 172.21.15.23:0/3373924255 conn(0x7f12e40fcc30 msgr2=0x7f12e40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 shutdown_connections 2024-03-28T15:51:00.333 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.332+0000 7f12e88dd640 1 -- 172.21.15.23:0/3373924255 wait complete. 2024-03-28T15:51:00.344 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-28T15:51:00.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1989410429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8104a70 msgr2=0x7fe3a8106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1989410429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8104a70 0x7fe3a8106e60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fe39c009900 tx=0x7fe39c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1989410429 shutdown_connections 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1989410429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a81073a0 0x7fe3a81097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1989410429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8104a70 0x7fe3a8106e60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1989410429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a8102140 0x7fe3a8104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1989410429 >> 172.21.15.23:0/1989410429 conn(0x7fe3a80fbb30 msgr2=0x7fe3a80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1989410429 shutdown_connections 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.453+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1989410429 wait complete. 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 Processor -- start 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 -- start start 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104a70 0x7fe3a8197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81073a0 0x7fe3a819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810d340 con 0x7fe3a8102140 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810d1c0 con 0x7fe3a81073a0 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3aff1b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810d4c0 con 0x7fe3a8104a70 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40274/0 (socket says 172.21.15.23:40274) 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 -- 172.21.15.23:0/1393872244 learned_addr learned my addr 172.21.15.23:0/1393872244 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3ae4b8640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81073a0 0x7fe3a819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3ad4b6640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104a70 0x7fe3a8197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 -- 172.21.15.23:0/1393872244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104a70 msgr2=0x7fe3a8197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104a70 0x7fe3a8197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 -- 172.21.15.23:0/1393872244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81073a0 msgr2=0x7fe3a819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81073a0 0x7fe3a819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.454+0000 7fe3adcb7640 1 -- 172.21.15.23:0/1393872244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a819f690 con 0x7fe3a8102140 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3adcb7640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe39800e9f0 tx=0x7fe39800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39800cd80 con 0x7fe3a8102140 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a819f980 con 0x7fe3a8102140 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe398010910 con 0x7fe3a8102140 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3980188f0 con 0x7fe3a8102140 2024-03-28T15:51:00.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.455+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a819fef0 con 0x7fe3a8102140 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.456+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a80fd620 con 0x7fe3a8102140 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.457+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe398010ac0 con 0x7fe3a8102140 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.458+0000 7fe3a6ffd640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.458+0000 7fe3ad4b6640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.458+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe3980cf410 con 0x7fe3a8102140 2024-03-28T15:51:00.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.459+0000 7fe3ad4b6640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 0x7fe3900a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe39c005ab0 tx=0x7fe39c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.464 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.464+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe39809bc20 con 0x7fe3a8102140 2024-03-28T15:51:00.679 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.679+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3a8061b70 con 0x7fe3900a2800 2024-03-28T15:51:00.684 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.684+0000 7fe3a6ffd640 1 -- 172.21.15.23:0/1393872244 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fe3a8061b70 con 0x7fe3900a2800 2024-03-28T15:51:00.684 INFO:teuthology.orchestra.run.smithi023.stdout:/volumes/qa/sv_1/221c8d1e-ed4e-48f2-bb79-dba7ddaab0a5 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 msgr2=0x7fe3900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 0x7fe3900a4cc0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe39c005ab0 tx=0x7fe39c005fd0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 msgr2=0x7fe3a8197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe39800e9f0 tx=0x7fe39800eeb0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 shutdown_connections 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3a81073a0 0x7fe3a819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe3a8104a70 0x7fe3a8197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 --2- 172.21.15.23:0/1393872244 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe3a8102140 0x7fe3a8197a70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 >> 172.21.15.23:0/1393872244 conn(0x7fe3a80fbb30 msgr2=0x7fe3a8103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 shutdown_connections 2024-03-28T15:51:00.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.686+0000 7fe3aff1b640 1 -- 172.21.15.23:0/1393872244 wait complete. 2024-03-28T15:51:00.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:51:00.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.802+0000 7f39db176640 1 -- 172.21.15.23:0/3768983909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d41008f0 msgr2=0x7f39d4100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.802+0000 7f39db176640 1 --2- 172.21.15.23:0/3768983909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d41008f0 0x7f39d4100d50 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f39c0009860 tx=0x7f39c002bb70 comp rx=0 tx=0).stop 2024-03-28T15:51:00.805 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.802+0000 7f39db176640 1 -- 172.21.15.23:0/3768983909 shutdown_connections 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.802+0000 7f39db176640 1 --2- 172.21.15.23:0/3768983909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d4101290 0x7f39d4101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.802+0000 7f39db176640 1 --2- 172.21.15.23:0/3768983909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d41008f0 0x7f39d4100d50 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 --2- 172.21.15.23:0/3768983909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d40fff90 0x7f39d41003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 -- 172.21.15.23:0/3768983909 >> 172.21.15.23:0/3768983909 conn(0x7f39d40fbb30 msgr2=0x7f39d40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 -- 172.21.15.23:0/3768983909 shutdown_connections 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 -- 172.21.15.23:0/3768983909 wait complete. 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 Processor -- start 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 -- start start 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 0x7f39d419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.803+0000 7f39db176640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d4101290 0x7f39d41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.806 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39db176640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d410f520 con 0x7f39d40fff90 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39db176640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d410f3a0 con 0x7f39d41008f0 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39db176640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d410f6a0 con 0x7f39d4101290 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d8f12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 0x7f39d419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d8f12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40294/0 (socket says 172.21.15.23:40294) 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d8f12640 1 -- 172.21.15.23:0/1901423909 learned_addr learned my addr 172.21.15.23:0/1901423909 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d9713640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d4101290 0x7f39d41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 -- 172.21.15.23:0/1901423909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d4101290 msgr2=0x7f39d41a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d4101290 0x7f39d41a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 -- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 msgr2=0x7f39d4199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 -- 172.21.15.23:0/1901423909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d41a17f0 con 0x7f39d41008f0 2024-03-28T15:51:00.807 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d8f12640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d3fff640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 0x7f39d419a110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39c002c0b0 tx=0x7f39c0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0041020 con 0x7f39d41008f0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d41a1a80 con 0x7f39d41008f0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39c002c9f0 con 0x7f39d41008f0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.804+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39c0039040 con 0x7f39d41008f0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.805+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d41a1fc0 con 0x7f39d41008f0 2024-03-28T15:51:00.808 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.806+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d410aec0 con 0x7f39d41008f0 2024-03-28T15:51:00.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.807+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f39c0044020 con 0x7f39d41008f0 2024-03-28T15:51:00.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.807+0000 7f39d1ffb640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 0x7f39c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:00.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.807+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f39c00f0ae0 con 0x7f39d41008f0 2024-03-28T15:51:00.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.808+0000 7f39d8f12640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 0x7f39c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:00.809 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.808+0000 7f39d8f12640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 0x7f39c40a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c8011500 tx=0x7f39c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:00.815 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:00.815+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39c004b020 con 0x7f39d41008f0 2024-03-28T15:51:01.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.044+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39d402d070 con 0x7f39d41008f0 2024-03-28T15:51:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.045+0000 7f39d1ffb640 1 -- 172.21.15.23:0/1901423909 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f39c00bda60 con 0x7f39d41008f0 2024-03-28T15:51:01.045 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:01.045 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:51:01.045 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 msgr2=0x7f39c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 0x7f39c40a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39c8011500 tx=0x7f39c8005f90 comp rx=0 tx=0).stop 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 msgr2=0x7f39d419a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 0x7f39d419a110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f39c002c0b0 tx=0x7f39c0032430 comp rx=0 tx=0).stop 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 shutdown_connections 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f39c40a2800 0x7f39c40a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39d4101290 0x7f39d41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39d41008f0 0x7f39d419a110 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 --2- 172.21.15.23:0/1901423909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39d40fff90 0x7f39d4199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 >> 172.21.15.23:0/1901423909 conn(0x7f39d40fbb30 msgr2=0x7f39d40fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 shutdown_connections 2024-03-28T15:51:01.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.047+0000 7f39db176640 1 -- 172.21.15.23:0/1901423909 wait complete. 2024-03-28T15:51:01.059 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 -- 172.21.15.23:0/4077428338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41005c0 msgr2=0x7fa7c41009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 --2- 172.21.15.23:0/4077428338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41005c0 0x7fa7c41009e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa7b4009860 tx=0x7fa7b402bb70 comp rx=0 tx=0).stop 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 -- 172.21.15.23:0/4077428338 shutdown_connections 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 --2- 172.21.15.23:0/4077428338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 --2- 172.21.15.23:0/4077428338 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41017b0 0x7fa7c4101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 --2- 172.21.15.23:0/4077428338 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41005c0 0x7fa7c41009e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 -- 172.21.15.23:0/4077428338 >> 172.21.15.23:0/4077428338 conn(0x7fa7c40fbb30 msgr2=0x7fa7c40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.164 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 -- 172.21.15.23:0/4077428338 shutdown_connections 2024-03-28T15:51:01.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.163+0000 7fa7ca659640 1 -- 172.21.15.23:0/4077428338 wait complete. 2024-03-28T15:51:01.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 Processor -- start 2024-03-28T15:51:01.165 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 -- start start 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41017b0 0x7fa7c4196e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c419de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c410c210 con 0x7fa7c41005c0 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c410c090 con 0x7fa7c4102170 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7ca659640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c410c390 con 0x7fa7c41017b0 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.164+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.167 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40312/0 (socket says 172.21.15.23:40312) 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 -- 172.21.15.23:0/36091194 learned_addr learned my addr 172.21.15.23:0/36091194 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c8bf6640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c419de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c37fe640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41017b0 0x7fa7c4196e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 -- 172.21.15.23:0/36091194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41017b0 msgr2=0x7fa7c4196e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41017b0 0x7fa7c4196e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 -- 172.21.15.23:0/36091194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 msgr2=0x7fa7c419de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c419de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 -- 172.21.15.23:0/36091194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c419e500 con 0x7fa7c41005c0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c8bf6640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c3fff640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa7b402c0b0 tx=0x7fa7b4032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4041020 con 0x7fa7c41005c0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c4068800 con 0x7fa7c41005c0 2024-03-28T15:51:01.168 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7b402c9f0 con 0x7fa7c41005c0 2024-03-28T15:51:01.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4039040 con 0x7fa7c41005c0 2024-03-28T15:51:01.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.165+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c4068cf0 con 0x7fa7c41005c0 2024-03-28T15:51:01.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.166+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c4068490 con 0x7fa7c41005c0 2024-03-28T15:51:01.169 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.167+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa7b4044020 con 0x7fa7c41005c0 2024-03-28T15:51:01.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.168+0000 7fa7c17fa640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 0x7fa7ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.168+0000 7fa7c37fe640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 0x7fa7ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.168+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa7b40f0920 con 0x7fa7c41005c0 2024-03-28T15:51:01.170 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.168+0000 7fa7c37fe640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 0x7fa7ac0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4197df0 tx=0x7fa7b8010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.173 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.173+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa7b404b020 con 0x7fa7c41005c0 2024-03-28T15:51:01.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.247+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa7b4032640 con 0x7fa7c41005c0 2024-03-28T15:51:01.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.365+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa7c4107420 con 0x7fa7c41005c0 2024-03-28T15:51:01.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.366+0000 7fa7c17fa640 1 -- 172.21.15.23:0/36091194 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7fa7c4107420 con 0x7fa7c41005c0 2024-03-28T15:51:01.367 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:01.367 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.368+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 msgr2=0x7fa7ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.368+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 0x7fa7ac0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4197df0 tx=0x7fa7b8010040 comp rx=0 tx=0).stop 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.368+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 msgr2=0x7fa7c41968e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.368+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa7b402c0b0 tx=0x7fa7b4032430 comp rx=0 tx=0).stop 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 shutdown_connections 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7ac0a2800 0x7fa7ac0a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa7c4102170 0x7fa7c419de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa7c41017b0 0x7fa7c4196e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 --2- 172.21.15.23:0/36091194 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa7c41005c0 0x7fa7c41968e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 >> 172.21.15.23:0/36091194 conn(0x7fa7c40fbb30 msgr2=0x7fa7c4104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 shutdown_connections 2024-03-28T15:51:01.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.369+0000 7fa7ca659640 1 -- 172.21.15.23:0/36091194 wait complete. 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:active 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.379 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:01 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1901423909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:51:01.381 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-28T15:51:01.381 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 -- 172.21.15.23:0/2203763195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb601073a0 msgr2=0x7fcb601097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 --2- 172.21.15.23:0/2203763195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb601073a0 0x7fcb601097f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009ca0 tx=0x7fcb5802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 -- 172.21.15.23:0/2203763195 shutdown_connections 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 --2- 172.21.15.23:0/2203763195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb601073a0 0x7fcb601097f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 --2- 172.21.15.23:0/2203763195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 --2- 172.21.15.23:0/2203763195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb60102140 0x7fcb60104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 -- 172.21.15.23:0/2203763195 >> 172.21.15.23:0/2203763195 conn(0x7fcb600fbb30 msgr2=0x7fcb600fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 -- 172.21.15.23:0/2203763195 shutdown_connections 2024-03-28T15:51:01.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.483+0000 7fcb6791b640 1 -- 172.21.15.23:0/2203763195 wait complete. 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 Processor -- start 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 -- start start 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb601073a0 0x7fcb6019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6010d360 con 0x7fcb60102140 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6010d1e0 con 0x7fcb60104a70 2024-03-28T15:51:01.487 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb6791b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6010d4e0 con 0x7fcb601073a0 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb656b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb656b7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40342/0 (socket says 172.21.15.23:40342) 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb656b7640 1 -- 172.21.15.23:0/1429045153 learned_addr learned my addr 172.21.15.23:0/1429045153 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.484+0000 7fcb64eb6640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb65eb8640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb601073a0 0x7fcb6019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 -- 172.21.15.23:0/1429045153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb601073a0 msgr2=0x7fcb6019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb601073a0 0x7fcb6019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 -- 172.21.15.23:0/1429045153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 msgr2=0x7fcb60197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 -- 172.21.15.23:0/1429045153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6019f5d0 con 0x7fcb60102140 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb64eb6640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb656b7640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00e9f0 tx=0x7fcb5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.488 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c00cd80 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6019f8c0 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb5c005500 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c010690 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.485+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6019fe10 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.486+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb600fd620 con 0x7fcb60102140 2024-03-28T15:51:01.489 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.487+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcb5c0046c0 con 0x7fcb60102140 2024-03-28T15:51:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.488+0000 7fcb567fc640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 0x7fcb4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.488+0000 7fcb64eb6640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 0x7fcb4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.488+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcb5c0cf220 con 0x7fcb60102140 2024-03-28T15:51:01.490 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.488+0000 7fcb64eb6640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 0x7fcb4c0a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb60198ec0 tx=0x7fcb50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.496 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.497+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcb5c09bf20 con 0x7fcb60102140 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:active 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:01 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1901423909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: from='client.14898 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] up:active 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:51:01.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:01.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:01 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1901423909' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:51:01.678 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.678+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcb60061ae0 con 0x7fcb4c0a2890 2024-03-28T15:51:01.691 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.691+0000 7fcb567fc640 1 -- 172.21.15.23:0/1429045153 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340598 (secure 0 0 0) 0x7fcb60061ae0 con 0x7fcb4c0a2890 2024-03-28T15:51:01.696 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:01.698 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T15:51:01.699 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 msgr2=0x7fcb4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 0x7fcb4c0a4d50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcb60198ec0 tx=0x7fcb50005f90 comp rx=0 tx=0).stop 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 msgr2=0x7fcb601979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c00e9f0 tx=0x7fcb5c00edf0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 shutdown_connections 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcb4c0a2890 0x7fcb4c0a4d50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcb601073a0 0x7fcb6019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb60104a70 0x7fcb60197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.700 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 --2- 172.21.15.23:0/1429045153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcb60102140 0x7fcb601979b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 >> 172.21.15.23:0/1429045153 conn(0x7fcb600fbb30 msgr2=0x7fcb60102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 shutdown_connections 2024-03-28T15:51:01.701 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.695+0000 7fcb6791b640 1 -- 172.21.15.23:0/1429045153 wait complete. 2024-03-28T15:51:01.706 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-28T15:51:00.374591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617187,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":329,"num_write_kb":2997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":264,"ondisk_log_size":264,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369244,"kb_used_data":11756,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572580,"statfs":{"total":1151940427776,"available":1151562321920,"internally_reserved":0,"allocated":12038144,"data_stored":7584983,"data_compressed":285357,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22320,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":7,"num_read_kb":9,"num_write":101,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003617"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241777+0000","last_change":"2024-03-28T15:50:32.417721+0000","last_active":"2024-03-28T15:50:39.241777+0000","last_peered":"2024-03-28T15:50:39.241777+0000","last_clean":"2024-03-28T15:50:39.241777+0000","last_became_active":"2024-03-28T15:50:32.417397+0000","last_became_peered":"2024-03-28T15:50:32.417397+0000","last_unstale":"2024-03-28T15:50:39.241777+0000","last_undegraded":"2024-03-28T15:50:39.241777+0000","last_fullsized":"2024-03-28T15:50:39.241777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:18.755144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336435+0000","last_change":"2024-03-28T15:50:32.419861+0000","last_active":"2024-03-28T15:50:39.336435+0000","last_peered":"2024-03-28T15:50:39.336435+0000","last_clean":"2024-03-28T15:50:39.336435+0000","last_became_active":"2024-03-28T15:50:32.419472+0000","last_became_peered":"2024-03-28T15:50:32.419472+0000","last_unstale":"2024-03-28T15:50:39.336435+0000","last_undegraded":"2024-03-28T15:50:39.336435+0000","last_fullsized":"2024-03-28T15:50:39.336435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:20.452492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336517+0000","last_change":"2024-03-28T15:50:32.425423+0000","last_active":"2024-03-28T15:50:39.336517+0000","last_peered":"2024-03-28T15:50:39.336517+0000","last_clean":"2024-03-28T15:50:39.336517+0000","last_became_active":"2024-03-28T15:50:32.423503+0000","last_became_peered":"2024-03-28T15:50:32.423503+0000","last_unstale":"2024-03-28T15:50:39.336517+0000","last_undegraded":"2024-03-28T15:50:39.336517+0000","last_fullsized":"2024-03-28T15:50:39.336517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:41.518085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825729+0000","last_change":"2024-03-28T15:50:32.426795+0000","last_active":"2024-03-28T15:50:37.825729+0000","last_peered":"2024-03-28T15:50:37.825729+0000","last_clean":"2024-03-28T15:50:37.825729+0000","last_became_active":"2024-03-28T15:50:32.426495+0000","last_became_peered":"2024-03-28T15:50:32.426495+0000","last_unstale":"2024-03-28T15:50:37.825729+0000","last_undegraded":"2024-03-28T15:50:37.825729+0000","last_fullsized":"2024-03-28T15:50:37.825729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:57:55.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.631922+0000","last_change":"2024-03-28T15:50:32.432258+0000","last_active":"2024-03-28T15:50:38.631922+0000","last_peered":"2024-03-28T15:50:38.631922+0000","last_clean":"2024-03-28T15:50:38.631922+0000","last_became_active":"2024-03-28T15:50:32.431840+0000","last_became_peered":"2024-03-28T15:50:32.431840+0000","last_unstale":"2024-03-28T15:50:38.631922+0000","last_undegraded":"2024-03-28T15:50:38.631922+0000","last_fullsized":"2024-03-28T15:50:38.631922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:30.878251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336063+0000","last_change":"2024-03-28T15:50:32.421362+0000","last_active":"2024-03-28T15:50:39.336063+0000","last_peered":"2024-03-28T15:50:39.336063+0000","last_clean":"2024-03-28T15:50:39.336063+0000","last_became_active":"2024-03-28T15:50:32.421039+0000","last_became_peered":"2024-03-28T15:50:32.421039+0000","last_unstale":"2024-03-28T15:50:39.336063+0000","last_undegraded":"2024-03-28T15:50:39.336063+0000","last_fullsized":"2024-03-28T15:50:39.336063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:01.590154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443642+0000","last_change":"2024-03-28T15:50:32.423923+0000","last_active":"2024-03-28T15:50:37.443642+0000","last_peered":"2024-03-28T15:50:37.443642+0000","last_clean":"2024-03-28T15:50:37.443642+0000","last_became_active":"2024-03-28T15:50:32.423147+0000","last_became_peered":"2024-03-28T15:50:32.423147+0000","last_unstale":"2024-03-28T15:50:37.443642+0000","last_undegraded":"2024-03-28T15:50:37.443642+0000","last_fullsized":"2024-03-28T15:50:37.443642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:42:18.526091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343359+0000","last_change":"2024-03-28T15:50:32.428936+0000","last_active":"2024-03-28T15:50:38.343359+0000","last_peered":"2024-03-28T15:50:38.343359+0000","last_clean":"2024-03-28T15:50:38.343359+0000","last_became_active":"2024-03-28T15:50:32.427438+0000","last_became_peered":"2024-03-28T15:50:32.427438+0000","last_unstale":"2024-03-28T15:50:38.343359+0000","last_undegraded":"2024-03-28T15:50:38.343359+0000","last_fullsized":"2024-03-28T15:50:38.343359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:20.788775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412582+0000","last_change":"2024-03-28T15:50:32.426691+0000","last_active":"2024-03-28T15:50:38.412582+0000","last_peered":"2024-03-28T15:50:38.412582+0000","last_clean":"2024-03-28T15:50:38.412582+0000","last_became_active":"2024-03-28T15:50:32.426408+0000","last_became_peered":"2024-03-28T15:50:32.426408+0000","last_unstale":"2024-03-28T15:50:38.412582+0000","last_undegraded":"2024-03-28T15:50:38.412582+0000","last_fullsized":"2024-03-28T15:50:38.412582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:09:45.911656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336399+0000","last_change":"2024-03-28T15:50:32.428322+0000","last_active":"2024-03-28T15:50:39.336399+0000","last_peered":"2024-03-28T15:50:39.336399+0000","last_clean":"2024-03-28T15:50:39.336399+0000","last_became_active":"2024-03-28T15:50:32.428001+0000","last_became_peered":"2024-03-28T15:50:32.428001+0000","last_unstale":"2024-03-28T15:50:39.336399+0000","last_undegraded":"2024-03-28T15:50:39.336399+0000","last_fullsized":"2024-03-28T15:50:39.336399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:00.795303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444091+0000","last_change":"2024-03-28T15:50:32.435072+0000","last_active":"2024-03-28T15:50:37.444091+0000","last_peered":"2024-03-28T15:50:37.444091+0000","last_clean":"2024-03-28T15:50:37.444091+0000","last_became_active":"2024-03-28T15:50:32.434757+0000","last_became_peered":"2024-03-28T15:50:32.434757+0000","last_unstale":"2024-03-28T15:50:37.444091+0000","last_undegraded":"2024-03-28T15:50:37.444091+0000","last_fullsized":"2024-03-28T15:50:37.444091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:34.357151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825725+0000","last_change":"2024-03-28T15:50:32.428312+0000","last_active":"2024-03-28T15:50:37.825725+0000","last_peered":"2024-03-28T15:50:37.825725+0000","last_clean":"2024-03-28T15:50:37.825725+0000","last_became_active":"2024-03-28T15:50:32.428020+0000","last_became_peered":"2024-03-28T15:50:32.428020+0000","last_unstale":"2024-03-28T15:50:37.825725+0000","last_undegraded":"2024-03-28T15:50:37.825725+0000","last_fullsized":"2024-03-28T15:50:37.825725+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:11.841606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441969+0000","last_change":"2024-03-28T15:50:32.422259+0000","last_active":"2024-03-28T15:50:37.441969+0000","last_peered":"2024-03-28T15:50:37.441969+0000","last_clean":"2024-03-28T15:50:37.441969+0000","last_became_active":"2024-03-28T15:50:32.421181+0000","last_became_peered":"2024-03-28T15:50:32.421181+0000","last_unstale":"2024-03-28T15:50:37.441969+0000","last_undegraded":"2024-03-28T15:50:37.441969+0000","last_fullsized":"2024-03-28T15:50:37.441969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:40.403642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335980+0000","last_change":"2024-03-28T15:50:32.430402+0000","last_active":"2024-03-28T15:50:39.335980+0000","last_peered":"2024-03-28T15:50:39.335980+0000","last_clean":"2024-03-28T15:50:39.335980+0000","last_became_active":"2024-03-28T15:50:32.429978+0000","last_became_peered":"2024-03-28T15:50:32.429978+0000","last_unstale":"2024-03-28T15:50:39.335980+0000","last_undegraded":"2024-03-28T15:50:39.335980+0000","last_fullsized":"2024-03-28T15:50:39.335980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:06.407373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335979+0000","last_change":"2024-03-28T15:50:32.429725+0000","last_active":"2024-03-28T15:50:39.335979+0000","last_peered":"2024-03-28T15:50:39.335979+0000","last_clean":"2024-03-28T15:50:39.335979+0000","last_became_active":"2024-03-28T15:50:32.429378+0000","last_became_peered":"2024-03-28T15:50:32.429378+0000","last_unstale":"2024-03-28T15:50:39.335979+0000","last_undegraded":"2024-03-28T15:50:39.335979+0000","last_fullsized":"2024-03-28T15:50:39.335979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:23.535587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412080+0000","last_change":"2024-03-28T15:50:32.429542+0000","last_active":"2024-03-28T15:50:38.412080+0000","last_peered":"2024-03-28T15:50:38.412080+0000","last_clean":"2024-03-28T15:50:38.412080+0000","last_became_active":"2024-03-28T15:50:32.427264+0000","last_became_peered":"2024-03-28T15:50:32.427264+0000","last_unstale":"2024-03-28T15:50:38.412080+0000","last_undegraded":"2024-03-28T15:50:38.412080+0000","last_fullsized":"2024-03-28T15:50:38.412080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.272678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241750+0000","last_change":"2024-03-28T15:50:32.418775+0000","last_active":"2024-03-28T15:50:39.241750+0000","last_peered":"2024-03-28T15:50:39.241750+0000","last_clean":"2024-03-28T15:50:39.241750+0000","last_became_active":"2024-03-28T15:50:32.418453+0000","last_became_peered":"2024-03-28T15:50:32.418453+0000","last_unstale":"2024-03-28T15:50:39.241750+0000","last_undegraded":"2024-03-28T15:50:39.241750+0000","last_fullsized":"2024-03-28T15:50:39.241750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:26.061231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336740+0000","last_change":"2024-03-28T15:50:32.429111+0000","last_active":"2024-03-28T15:50:39.336740+0000","last_peered":"2024-03-28T15:50:39.336740+0000","last_clean":"2024-03-28T15:50:39.336740+0000","last_became_active":"2024-03-28T15:50:32.428036+0000","last_became_peered":"2024-03-28T15:50:32.428036+0000","last_unstale":"2024-03-28T15:50:39.336740+0000","last_undegraded":"2024-03-28T15:50:39.336740+0000","last_fullsized":"2024-03-28T15:50:39.336740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:09.272224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336651+0000","last_change":"2024-03-28T15:50:32.422982+0000","last_active":"2024-03-28T15:50:39.336651+0000","last_peered":"2024-03-28T15:50:39.336651+0000","last_clean":"2024-03-28T15:50:39.336651+0000","last_became_active":"2024-03-28T15:50:32.422667+0000","last_became_peered":"2024-03-28T15:50:32.422667+0000","last_unstale":"2024-03-28T15:50:39.336651+0000","last_undegraded":"2024-03-28T15:50:39.336651+0000","last_fullsized":"2024-03-28T15:50:39.336651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:51.445603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336220+0000","last_change":"2024-03-28T15:50:36.462626+0000","last_active":"2024-03-28T15:50:39.336220+0000","last_peered":"2024-03-28T15:50:39.336220+0000","last_clean":"2024-03-28T15:50:39.336220+0000","last_became_active":"2024-03-28T15:50:36.462168+0000","last_became_peered":"2024-03-28T15:50:36.462168+0000","last_unstale":"2024-03-28T15:50:39.336220+0000","last_undegraded":"2024-03-28T15:50:39.336220+0000","last_fullsized":"2024-03-28T15:50:39.336220+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:59.676350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.228615+0000","last_change":"2024-03-28T15:50:31.425778+0000","last_active":"2024-03-28T15:50:59.228615+0000","last_peered":"2024-03-28T15:50:59.228615+0000","last_clean":"2024-03-28T15:50:59.228615+0000","last_became_active":"2024-03-28T15:50:31.424946+0000","last_became_peered":"2024-03-28T15:50:31.424946+0000","last_unstale":"2024-03-28T15:50:59.228615+0000","last_undegraded":"2024-03-28T15:50:59.228615+0000","last_fullsized":"2024-03-28T15:50:59.228615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:50.444459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343937+0000","last_change":"2024-03-28T15:50:32.431201+0000","last_active":"2024-03-28T15:50:38.343937+0000","last_peered":"2024-03-28T15:50:38.343937+0000","last_clean":"2024-03-28T15:50:38.343937+0000","last_became_active":"2024-03-28T15:50:32.428970+0000","last_became_peered":"2024-03-28T15:50:32.428970+0000","last_unstale":"2024-03-28T15:50:38.343937+0000","last_undegraded":"2024-03-28T15:50:38.343937+0000","last_fullsized":"2024-03-28T15:50:38.343937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:38:36.966552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825384+0000","last_change":"2024-03-28T15:50:36.457727+0000","last_active":"2024-03-28T15:50:37.825384+0000","last_peered":"2024-03-28T15:50:37.825384+0000","last_clean":"2024-03-28T15:50:37.825384+0000","last_became_active":"2024-03-28T15:50:36.457322+0000","last_became_peered":"2024-03-28T15:50:36.457322+0000","last_unstale":"2024-03-28T15:50:37.825384+0000","last_undegraded":"2024-03-28T15:50:37.825384+0000","last_fullsized":"2024-03-28T15:50:37.825384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:09.138564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.344135+0000","last_change":"2024-03-28T15:50:31.418890+0000","last_active":"2024-03-28T15:50:38.344135+0000","last_peered":"2024-03-28T15:50:38.344135+0000","last_clean":"2024-03-28T15:50:38.344135+0000","last_became_active":"2024-03-28T15:50:31.418445+0000","last_became_peered":"2024-03-28T15:50:31.418445+0000","last_unstale":"2024-03-28T15:50:38.344135+0000","last_undegraded":"2024-03-28T15:50:38.344135+0000","last_fullsized":"2024-03-28T15:50:38.344135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:15.157292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336615+0000","last_change":"2024-03-28T15:50:32.434362+0000","last_active":"2024-03-28T15:50:39.336615+0000","last_peered":"2024-03-28T15:50:39.336615+0000","last_clean":"2024-03-28T15:50:39.336615+0000","last_became_active":"2024-03-28T15:50:32.433577+0000","last_became_peered":"2024-03-28T15:50:32.433577+0000","last_unstale":"2024-03-28T15:50:39.336615+0000","last_undegraded":"2024-03-28T15:50:39.336615+0000","last_fullsized":"2024-03-28T15:50:39.336615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:13.424149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825458+0000","last_change":"2024-03-28T15:50:36.460311+0000","last_active":"2024-03-28T15:50:37.825458+0000","last_peered":"2024-03-28T15:50:37.825458+0000","last_clean":"2024-03-28T15:50:37.825458+0000","last_became_active":"2024-03-28T15:50:36.459934+0000","last_became_peered":"2024-03-28T15:50:36.459934+0000","last_unstale":"2024-03-28T15:50:37.825458+0000","last_undegraded":"2024-03-28T15:50:37.825458+0000","last_fullsized":"2024-03-28T15:50:37.825458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:11.299668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632405+0000","last_change":"2024-03-28T15:50:31.421609+0000","last_active":"2024-03-28T15:50:38.632405+0000","last_peered":"2024-03-28T15:50:38.632405+0000","last_clean":"2024-03-28T15:50:38.632405+0000","last_became_active":"2024-03-28T15:50:31.420979+0000","last_became_peered":"2024-03-28T15:50:31.420979+0000","last_unstale":"2024-03-28T15:50:38.632405+0000","last_undegraded":"2024-03-28T15:50:38.632405+0000","last_fullsized":"2024-03-28T15:50:38.632405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:44:42.974359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412290+0000","last_change":"2024-03-28T15:50:32.432909+0000","last_active":"2024-03-28T15:50:38.412290+0000","last_peered":"2024-03-28T15:50:38.412290+0000","last_clean":"2024-03-28T15:50:38.412290+0000","last_became_active":"2024-03-28T15:50:32.432167+0000","last_became_peered":"2024-03-28T15:50:32.432167+0000","last_unstale":"2024-03-28T15:50:38.412290+0000","last_undegraded":"2024-03-28T15:50:38.412290+0000","last_fullsized":"2024-03-28T15:50:38.412290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:20.168299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825370+0000","last_change":"2024-03-28T15:50:36.459184+0000","last_active":"2024-03-28T15:50:37.825370+0000","last_peered":"2024-03-28T15:50:37.825370+0000","last_clean":"2024-03-28T15:50:37.825370+0000","last_became_active":"2024-03-28T15:50:36.458807+0000","last_became_peered":"2024-03-28T15:50:36.458807+0000","last_unstale":"2024-03-28T15:50:37.825370+0000","last_undegraded":"2024-03-28T15:50:37.825370+0000","last_fullsized":"2024-03-28T15:50:37.825370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:51.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632049+0000","last_change":"2024-03-28T15:50:31.422344+0000","last_active":"2024-03-28T15:50:38.632049+0000","last_peered":"2024-03-28T15:50:38.632049+0000","last_clean":"2024-03-28T15:50:38.632049+0000","last_became_active":"2024-03-28T15:50:31.421418+0000","last_became_peered":"2024-03-28T15:50:31.421418+0000","last_unstale":"2024-03-28T15:50:38.632049+0000","last_undegraded":"2024-03-28T15:50:38.632049+0000","last_fullsized":"2024-03-28T15:50:38.632049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:17:11.341382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442855+0000","last_change":"2024-03-28T15:50:32.428599+0000","last_active":"2024-03-28T15:50:37.442855+0000","last_peered":"2024-03-28T15:50:37.442855+0000","last_clean":"2024-03-28T15:50:37.442855+0000","last_became_active":"2024-03-28T15:50:32.427823+0000","last_became_peered":"2024-03-28T15:50:32.427823+0000","last_unstale":"2024-03-28T15:50:37.442855+0000","last_undegraded":"2024-03-28T15:50:37.442855+0000","last_fullsized":"2024-03-28T15:50:37.442855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:18.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336488+0000","last_change":"2024-03-28T15:50:36.451717+0000","last_active":"2024-03-28T15:50:39.336488+0000","last_peered":"2024-03-28T15:50:39.336488+0000","last_clean":"2024-03-28T15:50:39.336488+0000","last_became_active":"2024-03-28T15:50:36.451217+0000","last_became_peered":"2024-03-28T15:50:36.451217+0000","last_unstale":"2024-03-28T15:50:39.336488+0000","last_undegraded":"2024-03-28T15:50:39.336488+0000","last_fullsized":"2024-03-28T15:50:39.336488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:29:05.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.562560+0000","last_change":"2024-03-28T15:50:31.419469+0000","last_active":"2024-03-28T15:50:56.562560+0000","last_peered":"2024-03-28T15:50:56.562560+0000","last_clean":"2024-03-28T15:50:56.562560+0000","last_became_active":"2024-03-28T15:50:31.419162+0000","last_became_peered":"2024-03-28T15:50:31.419162+0000","last_unstale":"2024-03-28T15:50:56.562560+0000","last_undegraded":"2024-03-28T15:50:56.562560+0000","last_fullsized":"2024-03-28T15:50:56.562560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:25.697846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632019+0000","last_change":"2024-03-28T15:50:32.422764+0000","last_active":"2024-03-28T15:50:38.632019+0000","last_peered":"2024-03-28T15:50:38.632019+0000","last_clean":"2024-03-28T15:50:38.632019+0000","last_became_active":"2024-03-28T15:50:32.422368+0000","last_became_peered":"2024-03-28T15:50:32.422368+0000","last_unstale":"2024-03-28T15:50:38.632019+0000","last_undegraded":"2024-03-28T15:50:38.632019+0000","last_fullsized":"2024-03-28T15:50:38.632019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:37:50.195055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336427+0000","last_change":"2024-03-28T15:50:36.453526+0000","last_active":"2024-03-28T15:50:39.336427+0000","last_peered":"2024-03-28T15:50:39.336427+0000","last_clean":"2024-03-28T15:50:39.336427+0000","last_became_active":"2024-03-28T15:50:36.452099+0000","last_became_peered":"2024-03-28T15:50:36.452099+0000","last_unstale":"2024-03-28T15:50:39.336427+0000","last_undegraded":"2024-03-28T15:50:39.336427+0000","last_fullsized":"2024-03-28T15:50:39.336427+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:55:06.379921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.577120+0000","last_change":"2024-03-28T15:50:31.423636+0000","last_active":"2024-03-28T15:50:58.577120+0000","last_peered":"2024-03-28T15:50:58.577120+0000","last_clean":"2024-03-28T15:50:58.577120+0000","last_became_active":"2024-03-28T15:50:31.423303+0000","last_became_peered":"2024-03-28T15:50:31.423303+0000","last_unstale":"2024-03-28T15:50:58.577120+0000","last_undegraded":"2024-03-28T15:50:58.577120+0000","last_fullsized":"2024-03-28T15:50:58.577120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:05:57.666490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442622+0000","last_change":"2024-03-28T15:50:32.428292+0000","last_active":"2024-03-28T15:50:37.442622+0000","last_peered":"2024-03-28T15:50:37.442622+0000","last_clean":"2024-03-28T15:50:37.442622+0000","last_became_active":"2024-03-28T15:50:32.427950+0000","last_became_peered":"2024-03-28T15:50:32.427950+0000","last_unstale":"2024-03-28T15:50:37.442622+0000","last_undegraded":"2024-03-28T15:50:37.442622+0000","last_fullsized":"2024-03-28T15:50:37.442622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:43.040690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241693+0000","last_change":"2024-03-28T15:50:36.461197+0000","last_active":"2024-03-28T15:50:39.241693+0000","last_peered":"2024-03-28T15:50:39.241693+0000","last_clean":"2024-03-28T15:50:39.241693+0000","last_became_active":"2024-03-28T15:50:36.460763+0000","last_became_peered":"2024-03-28T15:50:36.460763+0000","last_unstale":"2024-03-28T15:50:39.241693+0000","last_undegraded":"2024-03-28T15:50:39.241693+0000","last_fullsized":"2024-03-28T15:50:39.241693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:23.500309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567478+0000","last_change":"2024-03-28T15:50:31.417000+0000","last_active":"2024-03-28T15:50:57.567478+0000","last_peered":"2024-03-28T15:50:57.567478+0000","last_clean":"2024-03-28T15:50:57.567478+0000","last_became_active":"2024-03-28T15:50:31.416652+0000","last_became_peered":"2024-03-28T15:50:31.416652+0000","last_unstale":"2024-03-28T15:50:57.567478+0000","last_undegraded":"2024-03-28T15:50:57.567478+0000","last_fullsized":"2024-03-28T15:50:57.567478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:06:55.338636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343485+0000","last_change":"2024-03-28T15:50:32.432340+0000","last_active":"2024-03-28T15:50:38.343485+0000","last_peered":"2024-03-28T15:50:38.343485+0000","last_clean":"2024-03-28T15:50:38.343485+0000","last_became_active":"2024-03-28T15:50:32.432014+0000","last_became_peered":"2024-03-28T15:50:32.432014+0000","last_unstale":"2024-03-28T15:50:38.343485+0000","last_undegraded":"2024-03-28T15:50:38.343485+0000","last_fullsized":"2024-03-28T15:50:38.343485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:27.330100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336326+0000","last_change":"2024-03-28T15:50:36.460876+0000","last_active":"2024-03-28T15:50:39.336326+0000","last_peered":"2024-03-28T15:50:39.336326+0000","last_clean":"2024-03-28T15:50:39.336326+0000","last_became_active":"2024-03-28T15:50:36.460452+0000","last_became_peered":"2024-03-28T15:50:36.460452+0000","last_unstale":"2024-03-28T15:50:39.336326+0000","last_undegraded":"2024-03-28T15:50:39.336326+0000","last_fullsized":"2024-03-28T15:50:39.336326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:08:36.260861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442172+0000","last_change":"2024-03-28T15:50:31.417925+0000","last_active":"2024-03-28T15:50:37.442172+0000","last_peered":"2024-03-28T15:50:37.442172+0000","last_clean":"2024-03-28T15:50:37.442172+0000","last_became_active":"2024-03-28T15:50:31.417615+0000","last_became_peered":"2024-03-28T15:50:31.417615+0000","last_unstale":"2024-03-28T15:50:37.442172+0000","last_undegraded":"2024-03-28T15:50:37.442172+0000","last_fullsized":"2024-03-28T15:50:37.442172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:31.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335946+0000","last_change":"2024-03-28T15:50:32.423511+0000","last_active":"2024-03-28T15:50:39.335946+0000","last_peered":"2024-03-28T15:50:39.335946+0000","last_clean":"2024-03-28T15:50:39.335946+0000","last_became_active":"2024-03-28T15:50:32.423194+0000","last_became_peered":"2024-03-28T15:50:32.423194+0000","last_unstale":"2024-03-28T15:50:39.335946+0000","last_undegraded":"2024-03-28T15:50:39.335946+0000","last_fullsized":"2024-03-28T15:50:39.335946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:44.170488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343559+0000","last_change":"2024-03-28T15:50:36.465051+0000","last_active":"2024-03-28T15:50:38.343559+0000","last_peered":"2024-03-28T15:50:38.343559+0000","last_clean":"2024-03-28T15:50:38.343559+0000","last_became_active":"2024-03-28T15:50:36.464579+0000","last_became_peered":"2024-03-28T15:50:36.464579+0000","last_unstale":"2024-03-28T15:50:38.343559+0000","last_undegraded":"2024-03-28T15:50:38.343559+0000","last_fullsized":"2024-03-28T15:50:38.343559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:56.769485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.221788+0000","last_change":"2024-03-28T15:50:31.429484+0000","last_active":"2024-03-28T15:50:59.221788+0000","last_peered":"2024-03-28T15:50:59.221788+0000","last_clean":"2024-03-28T15:50:59.221788+0000","last_became_active":"2024-03-28T15:50:31.429206+0000","last_became_peered":"2024-03-28T15:50:31.429206+0000","last_unstale":"2024-03-28T15:50:59.221788+0000","last_undegraded":"2024-03-28T15:50:59.221788+0000","last_fullsized":"2024-03-28T15:50:59.221788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:05.599043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412345+0000","last_change":"2024-03-28T15:50:32.421813+0000","last_active":"2024-03-28T15:50:38.412345+0000","last_peered":"2024-03-28T15:50:38.412345+0000","last_clean":"2024-03-28T15:50:38.412345+0000","last_became_active":"2024-03-28T15:50:32.421510+0000","last_became_peered":"2024-03-28T15:50:32.421510+0000","last_unstale":"2024-03-28T15:50:38.412345+0000","last_undegraded":"2024-03-28T15:50:38.412345+0000","last_fullsized":"2024-03-28T15:50:38.412345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:06.314487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336426+0000","last_change":"2024-03-28T15:50:36.464148+0000","last_active":"2024-03-28T15:50:39.336426+0000","last_peered":"2024-03-28T15:50:39.336426+0000","last_clean":"2024-03-28T15:50:39.336426+0000","last_became_active":"2024-03-28T15:50:36.463696+0000","last_became_peered":"2024-03-28T15:50:36.463696+0000","last_unstale":"2024-03-28T15:50:39.336426+0000","last_undegraded":"2024-03-28T15:50:39.336426+0000","last_fullsized":"2024-03-28T15:50:39.336426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:57.733727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561791+0000","last_change":"2024-03-28T15:50:31.422969+0000","last_active":"2024-03-28T15:50:56.561791+0000","last_peered":"2024-03-28T15:50:56.561791+0000","last_clean":"2024-03-28T15:50:56.561791+0000","last_became_active":"2024-03-28T15:50:31.422664+0000","last_became_peered":"2024-03-28T15:50:31.422664+0000","last_unstale":"2024-03-28T15:50:56.561791+0000","last_undegraded":"2024-03-28T15:50:56.561791+0000","last_fullsized":"2024-03-28T15:50:56.561791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:24.469105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336877+0000","last_change":"2024-03-28T15:50:32.421415+0000","last_active":"2024-03-28T15:50:39.336877+0000","last_peered":"2024-03-28T15:50:39.336877+0000","last_clean":"2024-03-28T15:50:39.336877+0000","last_became_active":"2024-03-28T15:50:32.421100+0000","last_became_peered":"2024-03-28T15:50:32.421100+0000","last_unstale":"2024-03-28T15:50:39.336877+0000","last_undegraded":"2024-03-28T15:50:39.336877+0000","last_fullsized":"2024-03-28T15:50:39.336877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:20.477403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825710+0000","last_change":"2024-03-28T15:50:36.457721+0000","last_active":"2024-03-28T15:50:37.825710+0000","last_peered":"2024-03-28T15:50:37.825710+0000","last_clean":"2024-03-28T15:50:37.825710+0000","last_became_active":"2024-03-28T15:50:36.457028+0000","last_became_peered":"2024-03-28T15:50:36.457028+0000","last_unstale":"2024-03-28T15:50:37.825710+0000","last_undegraded":"2024-03-28T15:50:37.825710+0000","last_fullsized":"2024-03-28T15:50:37.825710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:47.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442820+0000","last_change":"2024-03-28T15:50:31.418265+0000","last_active":"2024-03-28T15:50:37.442820+0000","last_peered":"2024-03-28T15:50:37.442820+0000","last_clean":"2024-03-28T15:50:37.442820+0000","last_became_active":"2024-03-28T15:50:31.417944+0000","last_became_peered":"2024-03-28T15:50:31.417944+0000","last_unstale":"2024-03-28T15:50:37.442820+0000","last_undegraded":"2024-03-28T15:50:37.442820+0000","last_fullsized":"2024-03-28T15:50:37.442820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:58.633512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412512+0000","last_change":"2024-03-28T15:50:32.433180+0000","last_active":"2024-03-28T15:50:38.412512+0000","last_peered":"2024-03-28T15:50:38.412512+0000","last_clean":"2024-03-28T15:50:38.412512+0000","last_became_active":"2024-03-28T15:50:32.432598+0000","last_became_peered":"2024-03-28T15:50:32.432598+0000","last_unstale":"2024-03-28T15:50:38.412512+0000","last_undegraded":"2024-03-28T15:50:38.412512+0000","last_fullsized":"2024-03-28T15:50:38.412512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:16.051480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336275+0000","last_change":"2024-03-28T15:50:36.453471+0000","last_active":"2024-03-28T15:50:39.336275+0000","last_peered":"2024-03-28T15:50:39.336275+0000","last_clean":"2024-03-28T15:50:39.336275+0000","last_became_active":"2024-03-28T15:50:36.453070+0000","last_became_peered":"2024-03-28T15:50:36.453070+0000","last_unstale":"2024-03-28T15:50:39.336275+0000","last_undegraded":"2024-03-28T15:50:39.336275+0000","last_fullsized":"2024-03-28T15:50:39.336275+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:55.417086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'21","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.597314+0000","last_change":"2024-03-28T15:50:31.419585+0000","last_active":"2024-03-28T15:50:59.597314+0000","last_peered":"2024-03-28T15:50:59.597314+0000","last_clean":"2024-03-28T15:50:59.597314+0000","last_became_active":"2024-03-28T15:50:31.419241+0000","last_became_peered":"2024-03-28T15:50:31.419241+0000","last_unstale":"2024-03-28T15:50:59.597314+0000","last_undegraded":"2024-03-28T15:50:59.597314+0000","last_fullsized":"2024-03-28T15:50:59.597314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:34:17.438115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":23,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632228+0000","last_change":"2024-03-28T15:50:32.436526+0000","last_active":"2024-03-28T15:50:38.632228+0000","last_peered":"2024-03-28T15:50:38.632228+0000","last_clean":"2024-03-28T15:50:38.632228+0000","last_became_active":"2024-03-28T15:50:32.436211+0000","last_became_peered":"2024-03-28T15:50:32.436211+0000","last_unstale":"2024-03-28T15:50:38.632228+0000","last_undegraded":"2024-03-28T15:50:38.632228+0000","last_fullsized":"2024-03-28T15:50:38.632228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:04.424281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443204+0000","last_change":"2024-03-28T15:50:36.460308+0000","last_active":"2024-03-28T15:50:37.443204+0000","last_peered":"2024-03-28T15:50:37.443204+0000","last_clean":"2024-03-28T15:50:37.443204+0000","last_became_active":"2024-03-28T15:50:36.459417+0000","last_became_peered":"2024-03-28T15:50:36.459417+0000","last_unstale":"2024-03-28T15:50:37.443204+0000","last_undegraded":"2024-03-28T15:50:37.443204+0000","last_fullsized":"2024-03-28T15:50:37.443204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:22.022249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567379+0000","last_change":"2024-03-28T15:50:31.426829+0000","last_active":"2024-03-28T15:50:57.567379+0000","last_peered":"2024-03-28T15:50:57.567379+0000","last_clean":"2024-03-28T15:50:57.567379+0000","last_became_active":"2024-03-28T15:50:31.426434+0000","last_became_peered":"2024-03-28T15:50:31.426434+0000","last_unstale":"2024-03-28T15:50:57.567379+0000","last_undegraded":"2024-03-28T15:50:57.567379+0000","last_fullsized":"2024-03-28T15:50:57.567379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.567766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441951+0000","last_change":"2024-03-28T15:50:32.427927+0000","last_active":"2024-03-28T15:50:37.441951+0000","last_peered":"2024-03-28T15:50:37.441951+0000","last_clean":"2024-03-28T15:50:37.441951+0000","last_became_active":"2024-03-28T15:50:32.427609+0000","last_became_peered":"2024-03-28T15:50:32.427609+0000","last_unstale":"2024-03-28T15:50:37.441951+0000","last_undegraded":"2024-03-28T15:50:37.441951+0000","last_fullsized":"2024-03-28T15:50:37.441951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:49.120638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241396+0000","last_change":"2024-03-28T15:50:36.460355+0000","last_active":"2024-03-28T15:50:39.241396+0000","last_peered":"2024-03-28T15:50:39.241396+0000","last_clean":"2024-03-28T15:50:39.241396+0000","last_became_active":"2024-03-28T15:50:36.459935+0000","last_became_peered":"2024-03-28T15:50:36.459935+0000","last_unstale":"2024-03-28T15:50:39.241396+0000","last_undegraded":"2024-03-28T15:50:39.241396+0000","last_fullsized":"2024-03-28T15:50:39.241396+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.895844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561849+0000","last_change":"2024-03-28T15:50:31.417282+0000","last_active":"2024-03-28T15:50:56.561849+0000","last_peered":"2024-03-28T15:50:56.561849+0000","last_clean":"2024-03-28T15:50:56.561849+0000","last_became_active":"2024-03-28T15:50:31.416852+0000","last_became_peered":"2024-03-28T15:50:31.416852+0000","last_unstale":"2024-03-28T15:50:56.561849+0000","last_undegraded":"2024-03-28T15:50:56.561849+0000","last_fullsized":"2024-03-28T15:50:56.561849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:26:12.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444022+0000","last_change":"2024-03-28T15:50:32.430362+0000","last_active":"2024-03-28T15:50:37.444022+0000","last_peered":"2024-03-28T15:50:37.444022+0000","last_clean":"2024-03-28T15:50:37.444022+0000","last_became_active":"2024-03-28T15:50:32.429999+0000","last_became_peered":"2024-03-28T15:50:32.429999+0000","last_unstale":"2024-03-28T15:50:37.444022+0000","last_undegraded":"2024-03-28T15:50:37.444022+0000","last_fullsized":"2024-03-28T15:50:37.444022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:47.613322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336702+0000","last_change":"2024-03-28T15:50:36.463116+0000","last_active":"2024-03-28T15:50:39.336702+0000","last_peered":"2024-03-28T15:50:39.336702+0000","last_clean":"2024-03-28T15:50:39.336702+0000","last_became_active":"2024-03-28T15:50:36.462705+0000","last_became_peered":"2024-03-28T15:50:36.462705+0000","last_unstale":"2024-03-28T15:50:39.336702+0000","last_undegraded":"2024-03-28T15:50:39.336702+0000","last_fullsized":"2024-03-28T15:50:39.336702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:49.876404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.564767+0000","last_change":"2024-03-28T15:50:31.418595+0000","last_active":"2024-03-28T15:50:57.564767+0000","last_peered":"2024-03-28T15:50:57.564767+0000","last_clean":"2024-03-28T15:50:57.564767+0000","last_became_active":"2024-03-28T15:50:31.418185+0000","last_became_peered":"2024-03-28T15:50:31.418185+0000","last_unstale":"2024-03-28T15:50:57.564767+0000","last_undegraded":"2024-03-28T15:50:57.564767+0000","last_fullsized":"2024-03-28T15:50:57.564767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:40.246760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443807+0000","last_change":"2024-03-28T15:50:32.431867+0000","last_active":"2024-03-28T15:50:37.443807+0000","last_peered":"2024-03-28T15:50:37.443807+0000","last_clean":"2024-03-28T15:50:37.443807+0000","last_became_active":"2024-03-28T15:50:32.431552+0000","last_became_peered":"2024-03-28T15:50:32.431552+0000","last_unstale":"2024-03-28T15:50:37.443807+0000","last_undegraded":"2024-03-28T15:50:37.443807+0000","last_fullsized":"2024-03-28T15:50:37.443807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:20.849087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442219+0000","last_change":"2024-03-28T15:50:36.466785+0000","last_active":"2024-03-28T15:50:37.442219+0000","last_peered":"2024-03-28T15:50:37.442219+0000","last_clean":"2024-03-28T15:50:37.442219+0000","last_became_active":"2024-03-28T15:50:36.466356+0000","last_became_peered":"2024-03-28T15:50:36.466356+0000","last_unstale":"2024-03-28T15:50:37.442219+0000","last_undegraded":"2024-03-28T15:50:37.442219+0000","last_fullsized":"2024-03-28T15:50:37.442219+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:07.453802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561652+0000","last_change":"2024-03-28T15:50:31.417956+0000","last_active":"2024-03-28T15:50:56.561652+0000","last_peered":"2024-03-28T15:50:56.561652+0000","last_clean":"2024-03-28T15:50:56.561652+0000","last_became_active":"2024-03-28T15:50:31.417613+0000","last_became_peered":"2024-03-28T15:50:31.417613+0000","last_unstale":"2024-03-28T15:50:56.561652+0000","last_undegraded":"2024-03-28T15:50:56.561652+0000","last_fullsized":"2024-03-28T15:50:56.561652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:33.039685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442257+0000","last_change":"2024-03-28T15:50:32.426973+0000","last_active":"2024-03-28T15:50:37.442257+0000","last_peered":"2024-03-28T15:50:37.442257+0000","last_clean":"2024-03-28T15:50:37.442257+0000","last_became_active":"2024-03-28T15:50:32.426668+0000","last_became_peered":"2024-03-28T15:50:32.426668+0000","last_unstale":"2024-03-28T15:50:37.442257+0000","last_undegraded":"2024-03-28T15:50:37.442257+0000","last_fullsized":"2024-03-28T15:50:37.442257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:10:56.270066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442247+0000","last_change":"2024-03-28T15:50:36.463509+0000","last_active":"2024-03-28T15:50:37.442247+0000","last_peered":"2024-03-28T15:50:37.442247+0000","last_clean":"2024-03-28T15:50:37.442247+0000","last_became_active":"2024-03-28T15:50:36.462733+0000","last_became_peered":"2024-03-28T15:50:36.462733+0000","last_unstale":"2024-03-28T15:50:37.442247+0000","last_undegraded":"2024-03-28T15:50:37.442247+0000","last_fullsized":"2024-03-28T15:50:37.442247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:43:33.737012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632394+0000","last_change":"2024-03-28T15:50:31.420224+0000","last_active":"2024-03-28T15:50:38.632394+0000","last_peered":"2024-03-28T15:50:38.632394+0000","last_clean":"2024-03-28T15:50:38.632394+0000","last_became_active":"2024-03-28T15:50:31.419913+0000","last_became_peered":"2024-03-28T15:50:31.419913+0000","last_unstale":"2024-03-28T15:50:38.632394+0000","last_undegraded":"2024-03-28T15:50:38.632394+0000","last_fullsized":"2024-03-28T15:50:38.632394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:59:49.834297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241516+0000","last_change":"2024-03-28T15:50:32.425419+0000","last_active":"2024-03-28T15:50:39.241516+0000","last_peered":"2024-03-28T15:50:39.241516+0000","last_clean":"2024-03-28T15:50:39.241516+0000","last_became_active":"2024-03-28T15:50:32.425129+0000","last_became_peered":"2024-03-28T15:50:32.425129+0000","last_unstale":"2024-03-28T15:50:39.241516+0000","last_undegraded":"2024-03-28T15:50:39.241516+0000","last_fullsized":"2024-03-28T15:50:39.241516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:32.407032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632095+0000","last_change":"2024-03-28T15:50:36.460744+0000","last_active":"2024-03-28T15:50:38.632095+0000","last_peered":"2024-03-28T15:50:38.632095+0000","last_clean":"2024-03-28T15:50:38.632095+0000","last_became_active":"2024-03-28T15:50:36.460276+0000","last_became_peered":"2024-03-28T15:50:36.460276+0000","last_unstale":"2024-03-28T15:50:38.632095+0000","last_undegraded":"2024-03-28T15:50:38.632095+0000","last_fullsized":"2024-03-28T15:50:38.632095+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:12.600154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336564+0000","last_change":"2024-03-28T15:50:31.427718+0000","last_active":"2024-03-28T15:50:39.336564+0000","last_peered":"2024-03-28T15:50:39.336564+0000","last_clean":"2024-03-28T15:50:39.336564+0000","last_became_active":"2024-03-28T15:50:31.427414+0000","last_became_peered":"2024-03-28T15:50:31.427414+0000","last_unstale":"2024-03-28T15:50:39.336564+0000","last_undegraded":"2024-03-28T15:50:39.336564+0000","last_fullsized":"2024-03-28T15:50:39.336564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:24.636431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336159+0000","last_change":"2024-03-28T15:50:32.424409+0000","last_active":"2024-03-28T15:50:39.336159+0000","last_peered":"2024-03-28T15:50:39.336159+0000","last_clean":"2024-03-28T15:50:39.336159+0000","last_became_active":"2024-03-28T15:50:32.424108+0000","last_became_peered":"2024-03-28T15:50:32.424108+0000","last_unstale":"2024-03-28T15:50:39.336159+0000","last_undegraded":"2024-03-28T15:50:39.336159+0000","last_fullsized":"2024-03-28T15:50:39.336159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:59:23.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442262+0000","last_change":"2024-03-28T15:50:36.463533+0000","last_active":"2024-03-28T15:50:37.442262+0000","last_peered":"2024-03-28T15:50:37.442262+0000","last_clean":"2024-03-28T15:50:37.442262+0000","last_became_active":"2024-03-28T15:50:36.463047+0000","last_became_peered":"2024-03-28T15:50:36.463047+0000","last_unstale":"2024-03-28T15:50:37.442262+0000","last_undegraded":"2024-03-28T15:50:37.442262+0000","last_fullsized":"2024-03-28T15:50:37.442262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:52.072250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443761+0000","last_change":"2024-03-28T15:50:31.417073+0000","last_active":"2024-03-28T15:50:37.443761+0000","last_peered":"2024-03-28T15:50:37.443761+0000","last_clean":"2024-03-28T15:50:37.443761+0000","last_became_active":"2024-03-28T15:50:31.416755+0000","last_became_peered":"2024-03-28T15:50:31.416755+0000","last_unstale":"2024-03-28T15:50:37.443761+0000","last_undegraded":"2024-03-28T15:50:37.443761+0000","last_fullsized":"2024-03-28T15:50:37.443761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:45.887894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336211+0000","last_change":"2024-03-28T15:50:32.424912+0000","last_active":"2024-03-28T15:50:39.336211+0000","last_peered":"2024-03-28T15:50:39.336211+0000","last_clean":"2024-03-28T15:50:39.336211+0000","last_became_active":"2024-03-28T15:50:32.424609+0000","last_became_peered":"2024-03-28T15:50:32.424609+0000","last_unstale":"2024-03-28T15:50:39.336211+0000","last_undegraded":"2024-03-28T15:50:39.336211+0000","last_fullsized":"2024-03-28T15:50:39.336211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:37.539793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441937+0000","last_change":"2024-03-28T15:50:36.463968+0000","last_active":"2024-03-28T15:50:37.441937+0000","last_peered":"2024-03-28T15:50:37.441937+0000","last_clean":"2024-03-28T15:50:37.441937+0000","last_became_active":"2024-03-28T15:50:36.460697+0000","last_became_peered":"2024-03-28T15:50:36.460697+0000","last_unstale":"2024-03-28T15:50:37.441937+0000","last_undegraded":"2024-03-28T15:50:37.441937+0000","last_fullsized":"2024-03-28T15:50:37.441937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:00.726769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442247+0000","last_change":"2024-03-28T15:50:32.419783+0000","last_active":"2024-03-28T15:50:37.442247+0000","last_peered":"2024-03-28T15:50:37.442247+0000","last_clean":"2024-03-28T15:50:37.442247+0000","last_became_active":"2024-03-28T15:50:32.419404+0000","last_became_peered":"2024-03-28T15:50:32.419404+0000","last_unstale":"2024-03-28T15:50:37.442247+0000","last_undegraded":"2024-03-28T15:50:37.442247+0000","last_fullsized":"2024-03-28T15:50:37.442247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:42.437830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.568781+0000","last_change":"2024-03-28T15:50:31.414630+0000","last_active":"2024-03-28T15:50:57.568781+0000","last_peered":"2024-03-28T15:50:57.568781+0000","last_clean":"2024-03-28T15:50:57.568781+0000","last_became_active":"2024-03-28T15:50:31.414300+0000","last_became_peered":"2024-03-28T15:50:31.414300+0000","last_unstale":"2024-03-28T15:50:57.568781+0000","last_undegraded":"2024-03-28T15:50:57.568781+0000","last_fullsized":"2024-03-28T15:50:57.568781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:48.773649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443984+0000","last_change":"2024-03-28T15:50:36.462352+0000","last_active":"2024-03-28T15:50:37.443984+0000","last_peered":"2024-03-28T15:50:37.443984+0000","last_clean":"2024-03-28T15:50:37.443984+0000","last_became_active":"2024-03-28T15:50:36.462003+0000","last_became_peered":"2024-03-28T15:50:36.462003+0000","last_unstale":"2024-03-28T15:50:37.443984+0000","last_undegraded":"2024-03-28T15:50:37.443984+0000","last_fullsized":"2024-03-28T15:50:37.443984+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:42.830355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343849+0000","last_change":"2024-03-28T15:50:32.429086+0000","last_active":"2024-03-28T15:50:38.343849+0000","last_peered":"2024-03-28T15:50:38.343849+0000","last_clean":"2024-03-28T15:50:38.343849+0000","last_became_active":"2024-03-28T15:50:32.427761+0000","last_became_peered":"2024-03-28T15:50:32.427761+0000","last_unstale":"2024-03-28T15:50:38.343849+0000","last_undegraded":"2024-03-28T15:50:38.343849+0000","last_fullsized":"2024-03-28T15:50:38.343849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.883733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561837+0000","last_change":"2024-03-28T15:50:31.417017+0000","last_active":"2024-03-28T15:50:56.561837+0000","last_peered":"2024-03-28T15:50:56.561837+0000","last_clean":"2024-03-28T15:50:56.561837+0000","last_became_active":"2024-03-28T15:50:31.416688+0000","last_became_peered":"2024-03-28T15:50:31.416688+0000","last_unstale":"2024-03-28T15:50:56.561837+0000","last_undegraded":"2024-03-28T15:50:56.561837+0000","last_fullsized":"2024-03-28T15:50:56.561837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:39.364544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443656+0000","last_change":"2024-03-28T15:49:22.209051+0000","last_active":"2024-03-28T15:50:37.443656+0000","last_peered":"2024-03-28T15:50:37.443656+0000","last_clean":"2024-03-28T15:50:37.443656+0000","last_became_active":"2024-03-28T15:49:22.208633+0000","last_became_peered":"2024-03-28T15:49:22.208633+0000","last_unstale":"2024-03-28T15:50:37.443656+0000","last_undegraded":"2024-03-28T15:50:37.443656+0000","last_fullsized":"2024-03-28T15:50:37.443656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_clean_scrub_stamp":"2024-03-28T15:45:59.266253+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:08:41.194614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442933+0000","last_change":"2024-03-28T15:50:36.463514+0000","last_active":"2024-03-28T15:50:37.442933+0000","last_peered":"2024-03-28T15:50:37.442933+0000","last_clean":"2024-03-28T15:50:37.442933+0000","last_became_active":"2024-03-28T15:50:36.463110+0000","last_became_peered":"2024-03-28T15:50:36.463110+0000","last_unstale":"2024-03-28T15:50:37.442933+0000","last_undegraded":"2024-03-28T15:50:37.442933+0000","last_fullsized":"2024-03-28T15:50:37.442933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:30.481123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441412+0000","last_change":"2024-03-28T15:50:32.427016+0000","last_active":"2024-03-28T15:50:37.441412+0000","last_peered":"2024-03-28T15:50:37.441412+0000","last_clean":"2024-03-28T15:50:37.441412+0000","last_became_active":"2024-03-28T15:50:32.426519+0000","last_became_peered":"2024-03-28T15:50:32.426519+0000","last_unstale":"2024-03-28T15:50:37.441412+0000","last_undegraded":"2024-03-28T15:50:37.441412+0000","last_fullsized":"2024-03-28T15:50:37.441412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:27.650290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443828+0000","last_change":"2024-03-28T15:50:31.420871+0000","last_active":"2024-03-28T15:50:37.443828+0000","last_peered":"2024-03-28T15:50:37.443828+0000","last_clean":"2024-03-28T15:50:37.443828+0000","last_became_active":"2024-03-28T15:50:31.420418+0000","last_became_peered":"2024-03-28T15:50:31.420418+0000","last_unstale":"2024-03-28T15:50:37.443828+0000","last_undegraded":"2024-03-28T15:50:37.443828+0000","last_fullsized":"2024-03-28T15:50:37.443828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:57.615539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241505+0000","last_change":"2024-03-28T15:50:36.457759+0000","last_active":"2024-03-28T15:50:39.241505+0000","last_peered":"2024-03-28T15:50:39.241505+0000","last_clean":"2024-03-28T15:50:39.241505+0000","last_became_active":"2024-03-28T15:50:36.457248+0000","last_became_peered":"2024-03-28T15:50:36.457248+0000","last_unstale":"2024-03-28T15:50:39.241505+0000","last_undegraded":"2024-03-28T15:50:39.241505+0000","last_fullsized":"2024-03-28T15:50:39.241505+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:13:30.420180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632303+0000","last_change":"2024-03-28T15:50:31.428270+0000","last_active":"2024-03-28T15:50:38.632303+0000","last_peered":"2024-03-28T15:50:38.632303+0000","last_clean":"2024-03-28T15:50:38.632303+0000","last_became_active":"2024-03-28T15:50:31.427888+0000","last_became_peered":"2024-03-28T15:50:31.427888+0000","last_unstale":"2024-03-28T15:50:38.632303+0000","last_undegraded":"2024-03-28T15:50:38.632303+0000","last_fullsized":"2024-03-28T15:50:38.632303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:48:51.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442183+0000","last_change":"2024-03-28T15:50:32.416631+0000","last_active":"2024-03-28T15:50:37.442183+0000","last_peered":"2024-03-28T15:50:37.442183+0000","last_clean":"2024-03-28T15:50:37.442183+0000","last_became_active":"2024-03-28T15:50:32.416300+0000","last_became_peered":"2024-03-28T15:50:32.416300+0000","last_unstale":"2024-03-28T15:50:37.442183+0000","last_undegraded":"2024-03-28T15:50:37.442183+0000","last_fullsized":"2024-03-28T15:50:37.442183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:19.158562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343845+0000","last_change":"2024-03-28T15:50:36.465991+0000","last_active":"2024-03-28T15:50:38.343845+0000","last_peered":"2024-03-28T15:50:38.343845+0000","last_clean":"2024-03-28T15:50:38.343845+0000","last_became_active":"2024-03-28T15:50:36.465506+0000","last_became_peered":"2024-03-28T15:50:36.465506+0000","last_unstale":"2024-03-28T15:50:38.343845+0000","last_undegraded":"2024-03-28T15:50:38.343845+0000","last_fullsized":"2024-03-28T15:50:38.343845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:22.889068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.575837+0000","last_change":"2024-03-28T15:50:31.423894+0000","last_active":"2024-03-28T15:50:58.575837+0000","last_peered":"2024-03-28T15:50:58.575837+0000","last_clean":"2024-03-28T15:50:58.575837+0000","last_became_active":"2024-03-28T15:50:31.423562+0000","last_became_peered":"2024-03-28T15:50:31.423562+0000","last_unstale":"2024-03-28T15:50:58.575837+0000","last_undegraded":"2024-03-28T15:50:58.575837+0000","last_fullsized":"2024-03-28T15:50:58.575837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:55.065632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343744+0000","last_change":"2024-03-28T15:50:32.435132+0000","last_active":"2024-03-28T15:50:38.343744+0000","last_peered":"2024-03-28T15:50:38.343744+0000","last_clean":"2024-03-28T15:50:38.343744+0000","last_became_active":"2024-03-28T15:50:32.434523+0000","last_became_peered":"2024-03-28T15:50:32.434523+0000","last_unstale":"2024-03-28T15:50:38.343744+0000","last_undegraded":"2024-03-28T15:50:38.343744+0000","last_fullsized":"2024-03-28T15:50:38.343744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:35.363388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336027+0000","last_change":"2024-03-28T15:50:36.453501+0000","last_active":"2024-03-28T15:50:39.336027+0000","last_peered":"2024-03-28T15:50:39.336027+0000","last_clean":"2024-03-28T15:50:39.336027+0000","last_became_active":"2024-03-28T15:50:36.452121+0000","last_became_peered":"2024-03-28T15:50:36.452121+0000","last_unstale":"2024-03-28T15:50:39.336027+0000","last_undegraded":"2024-03-28T15:50:39.336027+0000","last_fullsized":"2024-03-28T15:50:39.336027+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:48:33.354233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.560926+0000","last_change":"2024-03-28T15:50:31.425645+0000","last_active":"2024-03-28T15:50:56.560926+0000","last_peered":"2024-03-28T15:50:56.560926+0000","last_clean":"2024-03-28T15:50:56.560926+0000","last_became_active":"2024-03-28T15:50:31.425332+0000","last_became_peered":"2024-03-28T15:50:31.425332+0000","last_unstale":"2024-03-28T15:50:56.560926+0000","last_undegraded":"2024-03-28T15:50:56.560926+0000","last_fullsized":"2024-03-28T15:50:56.560926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:16.221936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336521+0000","last_change":"2024-03-28T15:50:32.432602+0000","last_active":"2024-03-28T15:50:39.336521+0000","last_peered":"2024-03-28T15:50:39.336521+0000","last_clean":"2024-03-28T15:50:39.336521+0000","last_became_active":"2024-03-28T15:50:32.432303+0000","last_became_peered":"2024-03-28T15:50:32.432303+0000","last_unstale":"2024-03-28T15:50:39.336521+0000","last_undegraded":"2024-03-28T15:50:39.336521+0000","last_fullsized":"2024-03-28T15:50:39.336521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.524582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825616+0000","last_change":"2024-03-28T15:50:36.457991+0000","last_active":"2024-03-28T15:50:37.825616+0000","last_peered":"2024-03-28T15:50:37.825616+0000","last_clean":"2024-03-28T15:50:37.825616+0000","last_became_active":"2024-03-28T15:50:36.457380+0000","last_became_peered":"2024-03-28T15:50:36.457380+0000","last_unstale":"2024-03-28T15:50:37.825616+0000","last_undegraded":"2024-03-28T15:50:37.825616+0000","last_fullsized":"2024-03-28T15:50:37.825616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:19.866888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.559756+0000","last_change":"2024-03-28T15:50:31.418403+0000","last_active":"2024-03-28T15:50:56.559756+0000","last_peered":"2024-03-28T15:50:56.559756+0000","last_clean":"2024-03-28T15:50:56.559756+0000","last_became_active":"2024-03-28T15:50:31.418102+0000","last_became_peered":"2024-03-28T15:50:31.418102+0000","last_unstale":"2024-03-28T15:50:56.559756+0000","last_undegraded":"2024-03-28T15:50:56.559756+0000","last_fullsized":"2024-03-28T15:50:56.559756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:01:56.175404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.631808+0000","last_change":"2024-03-28T15:50:32.417331+0000","last_active":"2024-03-28T15:50:38.631808+0000","last_peered":"2024-03-28T15:50:38.631808+0000","last_clean":"2024-03-28T15:50:38.631808+0000","last_became_active":"2024-03-28T15:50:32.417026+0000","last_became_peered":"2024-03-28T15:50:32.417026+0000","last_unstale":"2024-03-28T15:50:38.631808+0000","last_undegraded":"2024-03-28T15:50:38.631808+0000","last_fullsized":"2024-03-28T15:50:38.631808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:24.081575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336959+0000","last_change":"2024-03-28T15:50:36.454877+0000","last_active":"2024-03-28T15:50:39.336959+0000","last_peered":"2024-03-28T15:50:39.336959+0000","last_clean":"2024-03-28T15:50:39.336959+0000","last_became_active":"2024-03-28T15:50:36.453802+0000","last_became_peered":"2024-03-28T15:50:36.453802+0000","last_unstale":"2024-03-28T15:50:39.336959+0000","last_undegraded":"2024-03-28T15:50:39.336959+0000","last_fullsized":"2024-03-28T15:50:39.336959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:56.479558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.575409+0000","last_change":"2024-03-28T15:50:31.425672+0000","last_active":"2024-03-28T15:50:58.575409+0000","last_peered":"2024-03-28T15:50:58.575409+0000","last_clean":"2024-03-28T15:50:58.575409+0000","last_became_active":"2024-03-28T15:50:31.425324+0000","last_became_peered":"2024-03-28T15:50:31.425324+0000","last_unstale":"2024-03-28T15:50:58.575409+0000","last_undegraded":"2024-03-28T15:50:58.575409+0000","last_fullsized":"2024-03-28T15:50:58.575409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:48:46.380962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241161+0000","last_change":"2024-03-28T15:50:32.424831+0000","last_active":"2024-03-28T15:50:39.241161+0000","last_peered":"2024-03-28T15:50:39.241161+0000","last_clean":"2024-03-28T15:50:39.241161+0000","last_became_active":"2024-03-28T15:50:32.423678+0000","last_became_peered":"2024-03-28T15:50:32.423678+0000","last_unstale":"2024-03-28T15:50:39.241161+0000","last_undegraded":"2024-03-28T15:50:39.241161+0000","last_fullsized":"2024-03-28T15:50:39.241161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:35:06.895634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825514+0000","last_change":"2024-03-28T15:50:36.460255+0000","last_active":"2024-03-28T15:50:37.825514+0000","last_peered":"2024-03-28T15:50:37.825514+0000","last_clean":"2024-03-28T15:50:37.825514+0000","last_became_active":"2024-03-28T15:50:36.459832+0000","last_became_peered":"2024-03-28T15:50:36.459832+0000","last_unstale":"2024-03-28T15:50:37.825514+0000","last_undegraded":"2024-03-28T15:50:37.825514+0000","last_fullsized":"2024-03-28T15:50:37.825514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:13.911621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336045+0000","last_change":"2024-03-28T15:50:31.430248+0000","last_active":"2024-03-28T15:50:39.336045+0000","last_peered":"2024-03-28T15:50:39.336045+0000","last_clean":"2024-03-28T15:50:39.336045+0000","last_became_active":"2024-03-28T15:50:31.429964+0000","last_became_peered":"2024-03-28T15:50:31.429964+0000","last_unstale":"2024-03-28T15:50:39.336045+0000","last_undegraded":"2024-03-28T15:50:39.336045+0000","last_fullsized":"2024-03-28T15:50:39.336045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:10:25.193102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343825+0000","last_change":"2024-03-28T15:50:32.431687+0000","last_active":"2024-03-28T15:50:38.343825+0000","last_peered":"2024-03-28T15:50:38.343825+0000","last_clean":"2024-03-28T15:50:38.343825+0000","last_became_active":"2024-03-28T15:50:32.431295+0000","last_became_peered":"2024-03-28T15:50:32.431295+0000","last_unstale":"2024-03-28T15:50:38.343825+0000","last_undegraded":"2024-03-28T15:50:38.343825+0000","last_fullsized":"2024-03-28T15:50:38.343825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:46:12.055577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825611+0000","last_change":"2024-03-28T15:50:36.459198+0000","last_active":"2024-03-28T15:50:37.825611+0000","last_peered":"2024-03-28T15:50:37.825611+0000","last_clean":"2024-03-28T15:50:37.825611+0000","last_became_active":"2024-03-28T15:50:36.458815+0000","last_became_peered":"2024-03-28T15:50:36.458815+0000","last_unstale":"2024-03-28T15:50:37.825611+0000","last_undegraded":"2024-03-28T15:50:37.825611+0000","last_fullsized":"2024-03-28T15:50:37.825611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:19.302434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561912+0000","last_change":"2024-03-28T15:50:31.421319+0000","last_active":"2024-03-28T15:50:56.561912+0000","last_peered":"2024-03-28T15:50:56.561912+0000","last_clean":"2024-03-28T15:50:56.561912+0000","last_became_active":"2024-03-28T15:50:31.420399+0000","last_became_peered":"2024-03-28T15:50:31.420399+0000","last_unstale":"2024-03-28T15:50:56.561912+0000","last_undegraded":"2024-03-28T15:50:56.561912+0000","last_fullsized":"2024-03-28T15:50:56.561912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:45.594801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443793+0000","last_change":"2024-03-28T15:50:32.436155+0000","last_active":"2024-03-28T15:50:37.443793+0000","last_peered":"2024-03-28T15:50:37.443793+0000","last_clean":"2024-03-28T15:50:37.443793+0000","last_became_active":"2024-03-28T15:50:32.435453+0000","last_became_peered":"2024-03-28T15:50:32.435453+0000","last_unstale":"2024-03-28T15:50:37.443793+0000","last_undegraded":"2024-03-28T15:50:37.443793+0000","last_fullsized":"2024-03-28T15:50:37.443793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:44.159744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412308+0000","last_change":"2024-03-28T15:50:36.456103+0000","last_active":"2024-03-28T15:50:38.412308+0000","last_peered":"2024-03-28T15:50:38.412308+0000","last_clean":"2024-03-28T15:50:38.412308+0000","last_became_active":"2024-03-28T15:50:36.455724+0000","last_became_peered":"2024-03-28T15:50:36.455724+0000","last_unstale":"2024-03-28T15:50:38.412308+0000","last_undegraded":"2024-03-28T15:50:38.412308+0000","last_fullsized":"2024-03-28T15:50:38.412308+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:30.874245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567315+0000","last_change":"2024-03-28T15:50:31.423648+0000","last_active":"2024-03-28T15:50:57.567315+0000","last_peered":"2024-03-28T15:50:57.567315+0000","last_clean":"2024-03-28T15:50:57.567315+0000","last_became_active":"2024-03-28T15:50:31.423343+0000","last_became_peered":"2024-03-28T15:50:31.423343+0000","last_unstale":"2024-03-28T15:50:57.567315+0000","last_undegraded":"2024-03-28T15:50:57.567315+0000","last_fullsized":"2024-03-28T15:50:57.567315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:22.475804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444118+0000","last_change":"2024-03-28T15:50:32.432943+0000","last_active":"2024-03-28T15:50:37.444118+0000","last_peered":"2024-03-28T15:50:37.444118+0000","last_clean":"2024-03-28T15:50:37.444118+0000","last_became_active":"2024-03-28T15:50:32.432535+0000","last_became_peered":"2024-03-28T15:50:32.432535+0000","last_unstale":"2024-03-28T15:50:37.444118+0000","last_undegraded":"2024-03-28T15:50:37.444118+0000","last_fullsized":"2024-03-28T15:50:37.444118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:20:17.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336101+0000","last_change":"2024-03-28T15:50:36.463152+0000","last_active":"2024-03-28T15:50:39.336101+0000","last_peered":"2024-03-28T15:50:39.336101+0000","last_clean":"2024-03-28T15:50:39.336101+0000","last_became_active":"2024-03-28T15:50:36.462771+0000","last_became_peered":"2024-03-28T15:50:36.462771+0000","last_unstale":"2024-03-28T15:50:39.336101+0000","last_undegraded":"2024-03-28T15:50:39.336101+0000","last_fullsized":"2024-03-28T15:50:39.336101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:22.958078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442903+0000","last_change":"2024-03-28T15:50:31.419043+0000","last_active":"2024-03-28T15:50:37.442903+0000","last_peered":"2024-03-28T15:50:37.442903+0000","last_clean":"2024-03-28T15:50:37.442903+0000","last_became_active":"2024-03-28T15:50:31.418722+0000","last_became_peered":"2024-03-28T15:50:31.418722+0000","last_unstale":"2024-03-28T15:50:37.442903+0000","last_undegraded":"2024-03-28T15:50:37.442903+0000","last_fullsized":"2024-03-28T15:50:37.442903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:10.512442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444051+0000","last_change":"2024-03-28T15:50:32.431708+0000","last_active":"2024-03-28T15:50:37.444051+0000","last_peered":"2024-03-28T15:50:37.444051+0000","last_clean":"2024-03-28T15:50:37.444051+0000","last_became_active":"2024-03-28T15:50:32.431356+0000","last_became_peered":"2024-03-28T15:50:32.431356+0000","last_unstale":"2024-03-28T15:50:37.444051+0000","last_undegraded":"2024-03-28T15:50:37.444051+0000","last_fullsized":"2024-03-28T15:50:37.444051+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:23:49.409548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442042+0000","last_change":"2024-03-28T15:50:36.455924+0000","last_active":"2024-03-28T15:50:37.442042+0000","last_peered":"2024-03-28T15:50:37.442042+0000","last_clean":"2024-03-28T15:50:37.442042+0000","last_became_active":"2024-03-28T15:50:36.455511+0000","last_became_peered":"2024-03-28T15:50:36.455511+0000","last_unstale":"2024-03-28T15:50:37.442042+0000","last_undegraded":"2024-03-28T15:50:37.442042+0000","last_fullsized":"2024-03-28T15:50:37.442042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:48.039313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442073+0000","last_change":"2024-03-28T15:50:31.418393+0000","last_active":"2024-03-28T15:50:37.442073+0000","last_peered":"2024-03-28T15:50:37.442073+0000","last_clean":"2024-03-28T15:50:37.442073+0000","last_became_active":"2024-03-28T15:50:31.418086+0000","last_became_peered":"2024-03-28T15:50:31.418086+0000","last_unstale":"2024-03-28T15:50:37.442073+0000","last_undegraded":"2024-03-28T15:50:37.442073+0000","last_fullsized":"2024-03-28T15:50:37.442073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:44.882002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444185+0000","last_change":"2024-03-28T15:50:32.429008+0000","last_active":"2024-03-28T15:50:37.444185+0000","last_peered":"2024-03-28T15:50:37.444185+0000","last_clean":"2024-03-28T15:50:37.444185+0000","last_became_active":"2024-03-28T15:50:32.428398+0000","last_became_peered":"2024-03-28T15:50:32.428398+0000","last_unstale":"2024-03-28T15:50:37.444185+0000","last_undegraded":"2024-03-28T15:50:37.444185+0000","last_fullsized":"2024-03-28T15:50:37.444185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:50.908576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241219+0000","last_change":"2024-03-28T15:50:32.424785+0000","last_active":"2024-03-28T15:50:39.241219+0000","last_peered":"2024-03-28T15:50:39.241219+0000","last_clean":"2024-03-28T15:50:39.241219+0000","last_became_active":"2024-03-28T15:50:32.424134+0000","last_became_peered":"2024-03-28T15:50:32.424134+0000","last_unstale":"2024-03-28T15:50:39.241219+0000","last_undegraded":"2024-03-28T15:50:39.241219+0000","last_fullsized":"2024-03-28T15:50:39.241219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:20.842423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336299+0000","last_change":"2024-03-28T15:50:32.426830+0000","last_active":"2024-03-28T15:50:39.336299+0000","last_peered":"2024-03-28T15:50:39.336299+0000","last_clean":"2024-03-28T15:50:39.336299+0000","last_became_active":"2024-03-28T15:50:32.426492+0000","last_became_peered":"2024-03-28T15:50:32.426492+0000","last_unstale":"2024-03-28T15:50:39.336299+0000","last_undegraded":"2024-03-28T15:50:39.336299+0000","last_fullsized":"2024-03-28T15:50:39.336299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:11.847280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335941+0000","last_change":"2024-03-28T15:50:32.425448+0000","last_active":"2024-03-28T15:50:39.335941+0000","last_peered":"2024-03-28T15:50:39.335941+0000","last_clean":"2024-03-28T15:50:39.335941+0000","last_became_active":"2024-03-28T15:50:32.425148+0000","last_became_peered":"2024-03-28T15:50:32.425148+0000","last_unstale":"2024-03-28T15:50:39.335941+0000","last_undegraded":"2024-03-28T15:50:39.335941+0000","last_fullsized":"2024-03-28T15:50:39.335941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:31.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336292+0000","last_change":"2024-03-28T15:50:32.433610+0000","last_active":"2024-03-28T15:50:39.336292+0000","last_peered":"2024-03-28T15:50:39.336292+0000","last_clean":"2024-03-28T15:50:39.336292+0000","last_became_active":"2024-03-28T15:50:32.433096+0000","last_became_peered":"2024-03-28T15:50:32.433096+0000","last_unstale":"2024-03-28T15:50:39.336292+0000","last_undegraded":"2024-03-28T15:50:39.336292+0000","last_fullsized":"2024-03-28T15:50:39.336292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:20.479797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442061+0000","last_change":"2024-03-28T15:50:32.428195+0000","last_active":"2024-03-28T15:50:37.442061+0000","last_peered":"2024-03-28T15:50:37.442061+0000","last_clean":"2024-03-28T15:50:37.442061+0000","last_became_active":"2024-03-28T15:50:32.427877+0000","last_became_peered":"2024-03-28T15:50:32.427877+0000","last_unstale":"2024-03-28T15:50:37.442061+0000","last_undegraded":"2024-03-28T15:50:37.442061+0000","last_fullsized":"2024-03-28T15:50:37.442061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:56.998688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241597+0000","last_change":"2024-03-28T15:50:32.420177+0000","last_active":"2024-03-28T15:50:39.241597+0000","last_peered":"2024-03-28T15:50:39.241597+0000","last_clean":"2024-03-28T15:50:39.241597+0000","last_became_active":"2024-03-28T15:50:32.419805+0000","last_became_peered":"2024-03-28T15:50:32.419805+0000","last_unstale":"2024-03-28T15:50:39.241597+0000","last_undegraded":"2024-03-28T15:50:39.241597+0000","last_fullsized":"2024-03-28T15:50:39.241597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:04:31.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336588+0000","last_change":"2024-03-28T15:50:32.417500+0000","last_active":"2024-03-28T15:50:39.336588+0000","last_peered":"2024-03-28T15:50:39.336588+0000","last_clean":"2024-03-28T15:50:39.336588+0000","last_became_active":"2024-03-28T15:50:32.416843+0000","last_became_peered":"2024-03-28T15:50:32.416843+0000","last_unstale":"2024-03-28T15:50:39.336588+0000","last_undegraded":"2024-03-28T15:50:39.336588+0000","last_fullsized":"2024-03-28T15:50:39.336588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:18:03.656816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343844+0000","last_change":"2024-03-28T15:50:32.428655+0000","last_active":"2024-03-28T15:50:38.343844+0000","last_peered":"2024-03-28T15:50:38.343844+0000","last_clean":"2024-03-28T15:50:38.343844+0000","last_became_active":"2024-03-28T15:50:32.428297+0000","last_became_peered":"2024-03-28T15:50:32.428297+0000","last_unstale":"2024-03-28T15:50:38.343844+0000","last_undegraded":"2024-03-28T15:50:38.343844+0000","last_fullsized":"2024-03-28T15:50:38.343844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:55:32.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442186+0000","last_change":"2024-03-28T15:50:32.422012+0000","last_active":"2024-03-28T15:50:37.442186+0000","last_peered":"2024-03-28T15:50:37.442186+0000","last_clean":"2024-03-28T15:50:37.442186+0000","last_became_active":"2024-03-28T15:50:32.421465+0000","last_became_peered":"2024-03-28T15:50:32.421465+0000","last_unstale":"2024-03-28T15:50:37.442186+0000","last_undegraded":"2024-03-28T15:50:37.442186+0000","last_fullsized":"2024-03-28T15:50:37.442186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:35.235917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.344169+0000","last_change":"2024-03-28T15:50:32.426624+0000","last_active":"2024-03-28T15:50:38.344169+0000","last_peered":"2024-03-28T15:50:38.344169+0000","last_clean":"2024-03-28T15:50:38.344169+0000","last_became_active":"2024-03-28T15:50:32.425307+0000","last_became_peered":"2024-03-28T15:50:32.425307+0000","last_unstale":"2024-03-28T15:50:38.344169+0000","last_undegraded":"2024-03-28T15:50:38.344169+0000","last_fullsized":"2024-03-28T15:50:38.344169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:05.332057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412474+0000","last_change":"2024-03-28T15:50:32.430274+0000","last_active":"2024-03-28T15:50:38.412474+0000","last_peered":"2024-03-28T15:50:38.412474+0000","last_clean":"2024-03-28T15:50:38.412474+0000","last_became_active":"2024-03-28T15:50:32.429147+0000","last_became_peered":"2024-03-28T15:50:32.429147+0000","last_unstale":"2024-03-28T15:50:38.412474+0000","last_undegraded":"2024-03-28T15:50:38.412474+0000","last_fullsized":"2024-03-28T15:50:38.412474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:19.135249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441703+0000","last_change":"2024-03-28T15:50:32.434933+0000","last_active":"2024-03-28T15:50:37.441703+0000","last_peered":"2024-03-28T15:50:37.441703+0000","last_clean":"2024-03-28T15:50:37.441703+0000","last_became_active":"2024-03-28T15:50:32.434612+0000","last_became_peered":"2024-03-28T15:50:32.434612+0000","last_unstale":"2024-03-28T15:50:37.441703+0000","last_undegraded":"2024-03-28T15:50:37.441703+0000","last_fullsized":"2024-03-28T15:50:37.441703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:41.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241467+0000","last_change":"2024-03-28T15:50:32.426181+0000","last_active":"2024-03-28T15:50:39.241467+0000","last_peered":"2024-03-28T15:50:39.241467+0000","last_clean":"2024-03-28T15:50:39.241467+0000","last_became_active":"2024-03-28T15:50:32.425866+0000","last_became_peered":"2024-03-28T15:50:32.425866+0000","last_unstale":"2024-03-28T15:50:39.241467+0000","last_undegraded":"2024-03-28T15:50:39.241467+0000","last_fullsized":"2024-03-28T15:50:39.241467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:42.519070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250168,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26819,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":7,"num_read_kb":9,"num_write":148,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":62418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":485554,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1095511,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481084,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480892,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1071216,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":492460,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":480960,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":484764,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":480550,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480910,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:51:01.712 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T15:51:01.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 -- 172.21.15.23:0/4084900250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f78100910 msgr2=0x7f8f78100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 --2- 172.21.15.23:0/4084900250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f78100910 0x7f8f78100d70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009900 tx=0x7f8f6802bcd0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 -- 172.21.15.23:0/4084900250 shutdown_connections 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 --2- 172.21.15.23:0/4084900250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 0x7f8f78101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 --2- 172.21.15.23:0/4084900250 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f78100910 0x7f8f78100d70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 --2- 172.21.15.23:0/4084900250 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f780fffb0 0x7f8f781003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 -- 172.21.15.23:0/4084900250 >> 172.21.15.23:0/4084900250 conn(0x7f8f780fbb30 msgr2=0x7f8f780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 -- 172.21.15.23:0/4084900250 shutdown_connections 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.850+0000 7f8f7f769640 1 -- 172.21.15.23:0/4084900250 wait complete. 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 Processor -- start 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 -- start start 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 0x7f8f7819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 0x7f8f781a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7810f520 con 0x7f8f780fffb0 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7810f3a0 con 0x7f8f781012b0 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7f769640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7810f6a0 con 0x7f8f78100910 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40372/0 (socket says 172.21.15.23:40372) 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 -- 172.21.15.23:0/1218501001 learned_addr learned my addr 172.21.15.23:0/1218501001 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7dd06640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 0x7f8f781a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7cd04640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 0x7f8f7819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 -- 172.21.15.23:0/1218501001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 msgr2=0x7f8f7819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 0x7f8f7819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 -- 172.21.15.23:0/1218501001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 msgr2=0x7f8f781a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:01.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 0x7f8f781a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 -- 172.21.15.23:0/1218501001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f781a1810 con 0x7f8f780fffb0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7cd04640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 0x7f8f7819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.851+0000 7f8f7d505640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400e9f0 tx=0x7f8f7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.852+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7400cd80 con 0x7f8f780fffb0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.852+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f781a1b00 con 0x7f8f780fffb0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.852+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f74010910 con 0x7f8f780fffb0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.852+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740188f0 con 0x7f8f780fffb0 2024-03-28T15:51:01.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.852+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f781a2020 con 0x7f8f780fffb0 2024-03-28T15:51:01.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.853+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7810aef0 con 0x7f8f780fffb0 2024-03-28T15:51:01.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.854+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8f74010ac0 con 0x7f8f780fffb0 2024-03-28T15:51:01.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.854+0000 7f8f6e7fc640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 0x7f8f640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:01.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.855+0000 7f8f7cd04640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 0x7f8f640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:01.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.855+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8f740cf1f0 con 0x7f8f780fffb0 2024-03-28T15:51:01.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.855+0000 7f8f7cd04640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 0x7f8f640a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006010 tx=0x7f8f6802c700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:01.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:01.859+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8f7409bf90 con 0x7f8f780fffb0 2024-03-28T15:51:02.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.040+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f78061ae0 con 0x7f8f640a2890 2024-03-28T15:51:02.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.050+0000 7f8f6e7fc640 1 -- 172.21.15.23:0/1218501001 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340598 (secure 0 0 0) 0x7f8f78061ae0 con 0x7f8f640a2890 2024-03-28T15:51:02.053 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:02.055 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 msgr2=0x7f8f640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 0x7f8f640a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8f68006010 tx=0x7f8f6802c700 comp rx=0 tx=0).stop 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 msgr2=0x7f8f78199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8f7400e9f0 tx=0x7f8f7400eeb0 comp rx=0 tx=0).stop 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 shutdown_connections 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8f640a2890 0x7f8f640a4d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f781012b0 0x7f8f781a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8f78100910 0x7f8f7819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:02.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 --2- 172.21.15.23:0/1218501001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8f780fffb0 0x7f8f78199bf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:02.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 >> 172.21.15.23:0/1218501001 conn(0x7f8f780fbb30 msgr2=0x7f8f780fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:02.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 shutdown_connections 2024-03-28T15:51:02.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:02.053+0000 7f8f7f769640 1 -- 172.21.15.23:0/1218501001 wait complete. 2024-03-28T15:51:02.064 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-03-28T15:51:00.374591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":617187,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":143,"num_read_kb":128,"num_write":329,"num_write_kb":2997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":264,"ondisk_log_size":264,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369244,"kb_used_data":11756,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572580,"statfs":{"total":1151940427776,"available":1151562321920,"internally_reserved":0,"allocated":12038144,"data_stored":7584983,"data_compressed":285357,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22320,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":7,"num_read_kb":9,"num_write":101,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003617"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241777+0000","last_change":"2024-03-28T15:50:32.417721+0000","last_active":"2024-03-28T15:50:39.241777+0000","last_peered":"2024-03-28T15:50:39.241777+0000","last_clean":"2024-03-28T15:50:39.241777+0000","last_became_active":"2024-03-28T15:50:32.417397+0000","last_became_peered":"2024-03-28T15:50:32.417397+0000","last_unstale":"2024-03-28T15:50:39.241777+0000","last_undegraded":"2024-03-28T15:50:39.241777+0000","last_fullsized":"2024-03-28T15:50:39.241777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:18.755144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336435+0000","last_change":"2024-03-28T15:50:32.419861+0000","last_active":"2024-03-28T15:50:39.336435+0000","last_peered":"2024-03-28T15:50:39.336435+0000","last_clean":"2024-03-28T15:50:39.336435+0000","last_became_active":"2024-03-28T15:50:32.419472+0000","last_became_peered":"2024-03-28T15:50:32.419472+0000","last_unstale":"2024-03-28T15:50:39.336435+0000","last_undegraded":"2024-03-28T15:50:39.336435+0000","last_fullsized":"2024-03-28T15:50:39.336435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:20.452492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336517+0000","last_change":"2024-03-28T15:50:32.425423+0000","last_active":"2024-03-28T15:50:39.336517+0000","last_peered":"2024-03-28T15:50:39.336517+0000","last_clean":"2024-03-28T15:50:39.336517+0000","last_became_active":"2024-03-28T15:50:32.423503+0000","last_became_peered":"2024-03-28T15:50:32.423503+0000","last_unstale":"2024-03-28T15:50:39.336517+0000","last_undegraded":"2024-03-28T15:50:39.336517+0000","last_fullsized":"2024-03-28T15:50:39.336517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:41.518085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825729+0000","last_change":"2024-03-28T15:50:32.426795+0000","last_active":"2024-03-28T15:50:37.825729+0000","last_peered":"2024-03-28T15:50:37.825729+0000","last_clean":"2024-03-28T15:50:37.825729+0000","last_became_active":"2024-03-28T15:50:32.426495+0000","last_became_peered":"2024-03-28T15:50:32.426495+0000","last_unstale":"2024-03-28T15:50:37.825729+0000","last_undegraded":"2024-03-28T15:50:37.825729+0000","last_fullsized":"2024-03-28T15:50:37.825729+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:57:55.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.631922+0000","last_change":"2024-03-28T15:50:32.432258+0000","last_active":"2024-03-28T15:50:38.631922+0000","last_peered":"2024-03-28T15:50:38.631922+0000","last_clean":"2024-03-28T15:50:38.631922+0000","last_became_active":"2024-03-28T15:50:32.431840+0000","last_became_peered":"2024-03-28T15:50:32.431840+0000","last_unstale":"2024-03-28T15:50:38.631922+0000","last_undegraded":"2024-03-28T15:50:38.631922+0000","last_fullsized":"2024-03-28T15:50:38.631922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:30.878251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336063+0000","last_change":"2024-03-28T15:50:32.421362+0000","last_active":"2024-03-28T15:50:39.336063+0000","last_peered":"2024-03-28T15:50:39.336063+0000","last_clean":"2024-03-28T15:50:39.336063+0000","last_became_active":"2024-03-28T15:50:32.421039+0000","last_became_peered":"2024-03-28T15:50:32.421039+0000","last_unstale":"2024-03-28T15:50:39.336063+0000","last_undegraded":"2024-03-28T15:50:39.336063+0000","last_fullsized":"2024-03-28T15:50:39.336063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:01.590154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443642+0000","last_change":"2024-03-28T15:50:32.423923+0000","last_active":"2024-03-28T15:50:37.443642+0000","last_peered":"2024-03-28T15:50:37.443642+0000","last_clean":"2024-03-28T15:50:37.443642+0000","last_became_active":"2024-03-28T15:50:32.423147+0000","last_became_peered":"2024-03-28T15:50:32.423147+0000","last_unstale":"2024-03-28T15:50:37.443642+0000","last_undegraded":"2024-03-28T15:50:37.443642+0000","last_fullsized":"2024-03-28T15:50:37.443642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:42:18.526091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343359+0000","last_change":"2024-03-28T15:50:32.428936+0000","last_active":"2024-03-28T15:50:38.343359+0000","last_peered":"2024-03-28T15:50:38.343359+0000","last_clean":"2024-03-28T15:50:38.343359+0000","last_became_active":"2024-03-28T15:50:32.427438+0000","last_became_peered":"2024-03-28T15:50:32.427438+0000","last_unstale":"2024-03-28T15:50:38.343359+0000","last_undegraded":"2024-03-28T15:50:38.343359+0000","last_fullsized":"2024-03-28T15:50:38.343359+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:20.788775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412582+0000","last_change":"2024-03-28T15:50:32.426691+0000","last_active":"2024-03-28T15:50:38.412582+0000","last_peered":"2024-03-28T15:50:38.412582+0000","last_clean":"2024-03-28T15:50:38.412582+0000","last_became_active":"2024-03-28T15:50:32.426408+0000","last_became_peered":"2024-03-28T15:50:32.426408+0000","last_unstale":"2024-03-28T15:50:38.412582+0000","last_undegraded":"2024-03-28T15:50:38.412582+0000","last_fullsized":"2024-03-28T15:50:38.412582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:09:45.911656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336399+0000","last_change":"2024-03-28T15:50:32.428322+0000","last_active":"2024-03-28T15:50:39.336399+0000","last_peered":"2024-03-28T15:50:39.336399+0000","last_clean":"2024-03-28T15:50:39.336399+0000","last_became_active":"2024-03-28T15:50:32.428001+0000","last_became_peered":"2024-03-28T15:50:32.428001+0000","last_unstale":"2024-03-28T15:50:39.336399+0000","last_undegraded":"2024-03-28T15:50:39.336399+0000","last_fullsized":"2024-03-28T15:50:39.336399+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:00.795303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444091+0000","last_change":"2024-03-28T15:50:32.435072+0000","last_active":"2024-03-28T15:50:37.444091+0000","last_peered":"2024-03-28T15:50:37.444091+0000","last_clean":"2024-03-28T15:50:37.444091+0000","last_became_active":"2024-03-28T15:50:32.434757+0000","last_became_peered":"2024-03-28T15:50:32.434757+0000","last_unstale":"2024-03-28T15:50:37.444091+0000","last_undegraded":"2024-03-28T15:50:37.444091+0000","last_fullsized":"2024-03-28T15:50:37.444091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:34.357151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825725+0000","last_change":"2024-03-28T15:50:32.428312+0000","last_active":"2024-03-28T15:50:37.825725+0000","last_peered":"2024-03-28T15:50:37.825725+0000","last_clean":"2024-03-28T15:50:37.825725+0000","last_became_active":"2024-03-28T15:50:32.428020+0000","last_became_peered":"2024-03-28T15:50:32.428020+0000","last_unstale":"2024-03-28T15:50:37.825725+0000","last_undegraded":"2024-03-28T15:50:37.825725+0000","last_fullsized":"2024-03-28T15:50:37.825725+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:11.841606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441969+0000","last_change":"2024-03-28T15:50:32.422259+0000","last_active":"2024-03-28T15:50:37.441969+0000","last_peered":"2024-03-28T15:50:37.441969+0000","last_clean":"2024-03-28T15:50:37.441969+0000","last_became_active":"2024-03-28T15:50:32.421181+0000","last_became_peered":"2024-03-28T15:50:32.421181+0000","last_unstale":"2024-03-28T15:50:37.441969+0000","last_undegraded":"2024-03-28T15:50:37.441969+0000","last_fullsized":"2024-03-28T15:50:37.441969+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:40.403642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335980+0000","last_change":"2024-03-28T15:50:32.430402+0000","last_active":"2024-03-28T15:50:39.335980+0000","last_peered":"2024-03-28T15:50:39.335980+0000","last_clean":"2024-03-28T15:50:39.335980+0000","last_became_active":"2024-03-28T15:50:32.429978+0000","last_became_peered":"2024-03-28T15:50:32.429978+0000","last_unstale":"2024-03-28T15:50:39.335980+0000","last_undegraded":"2024-03-28T15:50:39.335980+0000","last_fullsized":"2024-03-28T15:50:39.335980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:06.407373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335979+0000","last_change":"2024-03-28T15:50:32.429725+0000","last_active":"2024-03-28T15:50:39.335979+0000","last_peered":"2024-03-28T15:50:39.335979+0000","last_clean":"2024-03-28T15:50:39.335979+0000","last_became_active":"2024-03-28T15:50:32.429378+0000","last_became_peered":"2024-03-28T15:50:32.429378+0000","last_unstale":"2024-03-28T15:50:39.335979+0000","last_undegraded":"2024-03-28T15:50:39.335979+0000","last_fullsized":"2024-03-28T15:50:39.335979+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:36:23.535587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412080+0000","last_change":"2024-03-28T15:50:32.429542+0000","last_active":"2024-03-28T15:50:38.412080+0000","last_peered":"2024-03-28T15:50:38.412080+0000","last_clean":"2024-03-28T15:50:38.412080+0000","last_became_active":"2024-03-28T15:50:32.427264+0000","last_became_peered":"2024-03-28T15:50:32.427264+0000","last_unstale":"2024-03-28T15:50:38.412080+0000","last_undegraded":"2024-03-28T15:50:38.412080+0000","last_fullsized":"2024-03-28T15:50:38.412080+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.272678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241750+0000","last_change":"2024-03-28T15:50:32.418775+0000","last_active":"2024-03-28T15:50:39.241750+0000","last_peered":"2024-03-28T15:50:39.241750+0000","last_clean":"2024-03-28T15:50:39.241750+0000","last_became_active":"2024-03-28T15:50:32.418453+0000","last_became_peered":"2024-03-28T15:50:32.418453+0000","last_unstale":"2024-03-28T15:50:39.241750+0000","last_undegraded":"2024-03-28T15:50:39.241750+0000","last_fullsized":"2024-03-28T15:50:39.241750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:26.061231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336740+0000","last_change":"2024-03-28T15:50:32.429111+0000","last_active":"2024-03-28T15:50:39.336740+0000","last_peered":"2024-03-28T15:50:39.336740+0000","last_clean":"2024-03-28T15:50:39.336740+0000","last_became_active":"2024-03-28T15:50:32.428036+0000","last_became_peered":"2024-03-28T15:50:32.428036+0000","last_unstale":"2024-03-28T15:50:39.336740+0000","last_undegraded":"2024-03-28T15:50:39.336740+0000","last_fullsized":"2024-03-28T15:50:39.336740+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:09.272224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336651+0000","last_change":"2024-03-28T15:50:32.422982+0000","last_active":"2024-03-28T15:50:39.336651+0000","last_peered":"2024-03-28T15:50:39.336651+0000","last_clean":"2024-03-28T15:50:39.336651+0000","last_became_active":"2024-03-28T15:50:32.422667+0000","last_became_peered":"2024-03-28T15:50:32.422667+0000","last_unstale":"2024-03-28T15:50:39.336651+0000","last_undegraded":"2024-03-28T15:50:39.336651+0000","last_fullsized":"2024-03-28T15:50:39.336651+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:51.445603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336220+0000","last_change":"2024-03-28T15:50:36.462626+0000","last_active":"2024-03-28T15:50:39.336220+0000","last_peered":"2024-03-28T15:50:39.336220+0000","last_clean":"2024-03-28T15:50:39.336220+0000","last_became_active":"2024-03-28T15:50:36.462168+0000","last_became_peered":"2024-03-28T15:50:36.462168+0000","last_unstale":"2024-03-28T15:50:39.336220+0000","last_undegraded":"2024-03-28T15:50:39.336220+0000","last_fullsized":"2024-03-28T15:50:39.336220+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:59.676350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":36,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.228615+0000","last_change":"2024-03-28T15:50:31.425778+0000","last_active":"2024-03-28T15:50:59.228615+0000","last_peered":"2024-03-28T15:50:59.228615+0000","last_clean":"2024-03-28T15:50:59.228615+0000","last_became_active":"2024-03-28T15:50:31.424946+0000","last_became_peered":"2024-03-28T15:50:31.424946+0000","last_unstale":"2024-03-28T15:50:59.228615+0000","last_undegraded":"2024-03-28T15:50:59.228615+0000","last_fullsized":"2024-03-28T15:50:59.228615+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:50.444459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343937+0000","last_change":"2024-03-28T15:50:32.431201+0000","last_active":"2024-03-28T15:50:38.343937+0000","last_peered":"2024-03-28T15:50:38.343937+0000","last_clean":"2024-03-28T15:50:38.343937+0000","last_became_active":"2024-03-28T15:50:32.428970+0000","last_became_peered":"2024-03-28T15:50:32.428970+0000","last_unstale":"2024-03-28T15:50:38.343937+0000","last_undegraded":"2024-03-28T15:50:38.343937+0000","last_fullsized":"2024-03-28T15:50:38.343937+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:38:36.966552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825384+0000","last_change":"2024-03-28T15:50:36.457727+0000","last_active":"2024-03-28T15:50:37.825384+0000","last_peered":"2024-03-28T15:50:37.825384+0000","last_clean":"2024-03-28T15:50:37.825384+0000","last_became_active":"2024-03-28T15:50:36.457322+0000","last_became_peered":"2024-03-28T15:50:36.457322+0000","last_unstale":"2024-03-28T15:50:37.825384+0000","last_undegraded":"2024-03-28T15:50:37.825384+0000","last_fullsized":"2024-03-28T15:50:37.825384+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:09.138564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.344135+0000","last_change":"2024-03-28T15:50:31.418890+0000","last_active":"2024-03-28T15:50:38.344135+0000","last_peered":"2024-03-28T15:50:38.344135+0000","last_clean":"2024-03-28T15:50:38.344135+0000","last_became_active":"2024-03-28T15:50:31.418445+0000","last_became_peered":"2024-03-28T15:50:31.418445+0000","last_unstale":"2024-03-28T15:50:38.344135+0000","last_undegraded":"2024-03-28T15:50:38.344135+0000","last_fullsized":"2024-03-28T15:50:38.344135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:15.157292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336615+0000","last_change":"2024-03-28T15:50:32.434362+0000","last_active":"2024-03-28T15:50:39.336615+0000","last_peered":"2024-03-28T15:50:39.336615+0000","last_clean":"2024-03-28T15:50:39.336615+0000","last_became_active":"2024-03-28T15:50:32.433577+0000","last_became_peered":"2024-03-28T15:50:32.433577+0000","last_unstale":"2024-03-28T15:50:39.336615+0000","last_undegraded":"2024-03-28T15:50:39.336615+0000","last_fullsized":"2024-03-28T15:50:39.336615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:13.424149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825458+0000","last_change":"2024-03-28T15:50:36.460311+0000","last_active":"2024-03-28T15:50:37.825458+0000","last_peered":"2024-03-28T15:50:37.825458+0000","last_clean":"2024-03-28T15:50:37.825458+0000","last_became_active":"2024-03-28T15:50:36.459934+0000","last_became_peered":"2024-03-28T15:50:36.459934+0000","last_unstale":"2024-03-28T15:50:37.825458+0000","last_undegraded":"2024-03-28T15:50:37.825458+0000","last_fullsized":"2024-03-28T15:50:37.825458+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:11.299668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632405+0000","last_change":"2024-03-28T15:50:31.421609+0000","last_active":"2024-03-28T15:50:38.632405+0000","last_peered":"2024-03-28T15:50:38.632405+0000","last_clean":"2024-03-28T15:50:38.632405+0000","last_became_active":"2024-03-28T15:50:31.420979+0000","last_became_peered":"2024-03-28T15:50:31.420979+0000","last_unstale":"2024-03-28T15:50:38.632405+0000","last_undegraded":"2024-03-28T15:50:38.632405+0000","last_fullsized":"2024-03-28T15:50:38.632405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:44:42.974359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412290+0000","last_change":"2024-03-28T15:50:32.432909+0000","last_active":"2024-03-28T15:50:38.412290+0000","last_peered":"2024-03-28T15:50:38.412290+0000","last_clean":"2024-03-28T15:50:38.412290+0000","last_became_active":"2024-03-28T15:50:32.432167+0000","last_became_peered":"2024-03-28T15:50:32.432167+0000","last_unstale":"2024-03-28T15:50:38.412290+0000","last_undegraded":"2024-03-28T15:50:38.412290+0000","last_fullsized":"2024-03-28T15:50:38.412290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:20.168299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825370+0000","last_change":"2024-03-28T15:50:36.459184+0000","last_active":"2024-03-28T15:50:37.825370+0000","last_peered":"2024-03-28T15:50:37.825370+0000","last_clean":"2024-03-28T15:50:37.825370+0000","last_became_active":"2024-03-28T15:50:36.458807+0000","last_became_peered":"2024-03-28T15:50:36.458807+0000","last_unstale":"2024-03-28T15:50:37.825370+0000","last_undegraded":"2024-03-28T15:50:37.825370+0000","last_fullsized":"2024-03-28T15:50:37.825370+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:51.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632049+0000","last_change":"2024-03-28T15:50:31.422344+0000","last_active":"2024-03-28T15:50:38.632049+0000","last_peered":"2024-03-28T15:50:38.632049+0000","last_clean":"2024-03-28T15:50:38.632049+0000","last_became_active":"2024-03-28T15:50:31.421418+0000","last_became_peered":"2024-03-28T15:50:31.421418+0000","last_unstale":"2024-03-28T15:50:38.632049+0000","last_undegraded":"2024-03-28T15:50:38.632049+0000","last_fullsized":"2024-03-28T15:50:38.632049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:17:11.341382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442855+0000","last_change":"2024-03-28T15:50:32.428599+0000","last_active":"2024-03-28T15:50:37.442855+0000","last_peered":"2024-03-28T15:50:37.442855+0000","last_clean":"2024-03-28T15:50:37.442855+0000","last_became_active":"2024-03-28T15:50:32.427823+0000","last_became_peered":"2024-03-28T15:50:32.427823+0000","last_unstale":"2024-03-28T15:50:37.442855+0000","last_undegraded":"2024-03-28T15:50:37.442855+0000","last_fullsized":"2024-03-28T15:50:37.442855+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:18.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336488+0000","last_change":"2024-03-28T15:50:36.451717+0000","last_active":"2024-03-28T15:50:39.336488+0000","last_peered":"2024-03-28T15:50:39.336488+0000","last_clean":"2024-03-28T15:50:39.336488+0000","last_became_active":"2024-03-28T15:50:36.451217+0000","last_became_peered":"2024-03-28T15:50:36.451217+0000","last_unstale":"2024-03-28T15:50:39.336488+0000","last_undegraded":"2024-03-28T15:50:39.336488+0000","last_fullsized":"2024-03-28T15:50:39.336488+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:29:05.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"90'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.562560+0000","last_change":"2024-03-28T15:50:31.419469+0000","last_active":"2024-03-28T15:50:56.562560+0000","last_peered":"2024-03-28T15:50:56.562560+0000","last_clean":"2024-03-28T15:50:56.562560+0000","last_became_active":"2024-03-28T15:50:31.419162+0000","last_became_peered":"2024-03-28T15:50:31.419162+0000","last_unstale":"2024-03-28T15:50:56.562560+0000","last_undegraded":"2024-03-28T15:50:56.562560+0000","last_fullsized":"2024-03-28T15:50:56.562560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:25.697846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632019+0000","last_change":"2024-03-28T15:50:32.422764+0000","last_active":"2024-03-28T15:50:38.632019+0000","last_peered":"2024-03-28T15:50:38.632019+0000","last_clean":"2024-03-28T15:50:38.632019+0000","last_became_active":"2024-03-28T15:50:32.422368+0000","last_became_peered":"2024-03-28T15:50:32.422368+0000","last_unstale":"2024-03-28T15:50:38.632019+0000","last_undegraded":"2024-03-28T15:50:38.632019+0000","last_fullsized":"2024-03-28T15:50:38.632019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:37:50.195055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336427+0000","last_change":"2024-03-28T15:50:36.453526+0000","last_active":"2024-03-28T15:50:39.336427+0000","last_peered":"2024-03-28T15:50:39.336427+0000","last_clean":"2024-03-28T15:50:39.336427+0000","last_became_active":"2024-03-28T15:50:36.452099+0000","last_became_peered":"2024-03-28T15:50:36.452099+0000","last_unstale":"2024-03-28T15:50:39.336427+0000","last_undegraded":"2024-03-28T15:50:39.336427+0000","last_fullsized":"2024-03-28T15:50:39.336427+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:55:06.379921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.577120+0000","last_change":"2024-03-28T15:50:31.423636+0000","last_active":"2024-03-28T15:50:58.577120+0000","last_peered":"2024-03-28T15:50:58.577120+0000","last_clean":"2024-03-28T15:50:58.577120+0000","last_became_active":"2024-03-28T15:50:31.423303+0000","last_became_peered":"2024-03-28T15:50:31.423303+0000","last_unstale":"2024-03-28T15:50:58.577120+0000","last_undegraded":"2024-03-28T15:50:58.577120+0000","last_fullsized":"2024-03-28T15:50:58.577120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:05:57.666490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442622+0000","last_change":"2024-03-28T15:50:32.428292+0000","last_active":"2024-03-28T15:50:37.442622+0000","last_peered":"2024-03-28T15:50:37.442622+0000","last_clean":"2024-03-28T15:50:37.442622+0000","last_became_active":"2024-03-28T15:50:32.427950+0000","last_became_peered":"2024-03-28T15:50:32.427950+0000","last_unstale":"2024-03-28T15:50:37.442622+0000","last_undegraded":"2024-03-28T15:50:37.442622+0000","last_fullsized":"2024-03-28T15:50:37.442622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:43.040690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241693+0000","last_change":"2024-03-28T15:50:36.461197+0000","last_active":"2024-03-28T15:50:39.241693+0000","last_peered":"2024-03-28T15:50:39.241693+0000","last_clean":"2024-03-28T15:50:39.241693+0000","last_became_active":"2024-03-28T15:50:36.460763+0000","last_became_peered":"2024-03-28T15:50:36.460763+0000","last_unstale":"2024-03-28T15:50:39.241693+0000","last_undegraded":"2024-03-28T15:50:39.241693+0000","last_fullsized":"2024-03-28T15:50:39.241693+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:23.500309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567478+0000","last_change":"2024-03-28T15:50:31.417000+0000","last_active":"2024-03-28T15:50:57.567478+0000","last_peered":"2024-03-28T15:50:57.567478+0000","last_clean":"2024-03-28T15:50:57.567478+0000","last_became_active":"2024-03-28T15:50:31.416652+0000","last_became_peered":"2024-03-28T15:50:31.416652+0000","last_unstale":"2024-03-28T15:50:57.567478+0000","last_undegraded":"2024-03-28T15:50:57.567478+0000","last_fullsized":"2024-03-28T15:50:57.567478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:06:55.338636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343485+0000","last_change":"2024-03-28T15:50:32.432340+0000","last_active":"2024-03-28T15:50:38.343485+0000","last_peered":"2024-03-28T15:50:38.343485+0000","last_clean":"2024-03-28T15:50:38.343485+0000","last_became_active":"2024-03-28T15:50:32.432014+0000","last_became_peered":"2024-03-28T15:50:32.432014+0000","last_unstale":"2024-03-28T15:50:38.343485+0000","last_undegraded":"2024-03-28T15:50:38.343485+0000","last_fullsized":"2024-03-28T15:50:38.343485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:27:27.330100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336326+0000","last_change":"2024-03-28T15:50:36.460876+0000","last_active":"2024-03-28T15:50:39.336326+0000","last_peered":"2024-03-28T15:50:39.336326+0000","last_clean":"2024-03-28T15:50:39.336326+0000","last_became_active":"2024-03-28T15:50:36.460452+0000","last_became_peered":"2024-03-28T15:50:36.460452+0000","last_unstale":"2024-03-28T15:50:39.336326+0000","last_undegraded":"2024-03-28T15:50:39.336326+0000","last_fullsized":"2024-03-28T15:50:39.336326+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:08:36.260861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442172+0000","last_change":"2024-03-28T15:50:31.417925+0000","last_active":"2024-03-28T15:50:37.442172+0000","last_peered":"2024-03-28T15:50:37.442172+0000","last_clean":"2024-03-28T15:50:37.442172+0000","last_became_active":"2024-03-28T15:50:31.417615+0000","last_became_peered":"2024-03-28T15:50:31.417615+0000","last_unstale":"2024-03-28T15:50:37.442172+0000","last_undegraded":"2024-03-28T15:50:37.442172+0000","last_fullsized":"2024-03-28T15:50:37.442172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:31.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335946+0000","last_change":"2024-03-28T15:50:32.423511+0000","last_active":"2024-03-28T15:50:39.335946+0000","last_peered":"2024-03-28T15:50:39.335946+0000","last_clean":"2024-03-28T15:50:39.335946+0000","last_became_active":"2024-03-28T15:50:32.423194+0000","last_became_peered":"2024-03-28T15:50:32.423194+0000","last_unstale":"2024-03-28T15:50:39.335946+0000","last_undegraded":"2024-03-28T15:50:39.335946+0000","last_fullsized":"2024-03-28T15:50:39.335946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:44.170488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343559+0000","last_change":"2024-03-28T15:50:36.465051+0000","last_active":"2024-03-28T15:50:38.343559+0000","last_peered":"2024-03-28T15:50:38.343559+0000","last_clean":"2024-03-28T15:50:38.343559+0000","last_became_active":"2024-03-28T15:50:36.464579+0000","last_became_peered":"2024-03-28T15:50:36.464579+0000","last_unstale":"2024-03-28T15:50:38.343559+0000","last_undegraded":"2024-03-28T15:50:38.343559+0000","last_fullsized":"2024-03-28T15:50:38.343559+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:56.769485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"90'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.221788+0000","last_change":"2024-03-28T15:50:31.429484+0000","last_active":"2024-03-28T15:50:59.221788+0000","last_peered":"2024-03-28T15:50:59.221788+0000","last_clean":"2024-03-28T15:50:59.221788+0000","last_became_active":"2024-03-28T15:50:31.429206+0000","last_became_peered":"2024-03-28T15:50:31.429206+0000","last_unstale":"2024-03-28T15:50:59.221788+0000","last_undegraded":"2024-03-28T15:50:59.221788+0000","last_fullsized":"2024-03-28T15:50:59.221788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:05.599043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412345+0000","last_change":"2024-03-28T15:50:32.421813+0000","last_active":"2024-03-28T15:50:38.412345+0000","last_peered":"2024-03-28T15:50:38.412345+0000","last_clean":"2024-03-28T15:50:38.412345+0000","last_became_active":"2024-03-28T15:50:32.421510+0000","last_became_peered":"2024-03-28T15:50:32.421510+0000","last_unstale":"2024-03-28T15:50:38.412345+0000","last_undegraded":"2024-03-28T15:50:38.412345+0000","last_fullsized":"2024-03-28T15:50:38.412345+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:06.314487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336426+0000","last_change":"2024-03-28T15:50:36.464148+0000","last_active":"2024-03-28T15:50:39.336426+0000","last_peered":"2024-03-28T15:50:39.336426+0000","last_clean":"2024-03-28T15:50:39.336426+0000","last_became_active":"2024-03-28T15:50:36.463696+0000","last_became_peered":"2024-03-28T15:50:36.463696+0000","last_unstale":"2024-03-28T15:50:39.336426+0000","last_undegraded":"2024-03-28T15:50:39.336426+0000","last_fullsized":"2024-03-28T15:50:39.336426+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:57.733727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561791+0000","last_change":"2024-03-28T15:50:31.422969+0000","last_active":"2024-03-28T15:50:56.561791+0000","last_peered":"2024-03-28T15:50:56.561791+0000","last_clean":"2024-03-28T15:50:56.561791+0000","last_became_active":"2024-03-28T15:50:31.422664+0000","last_became_peered":"2024-03-28T15:50:31.422664+0000","last_unstale":"2024-03-28T15:50:56.561791+0000","last_undegraded":"2024-03-28T15:50:56.561791+0000","last_fullsized":"2024-03-28T15:50:56.561791+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:24.469105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336877+0000","last_change":"2024-03-28T15:50:32.421415+0000","last_active":"2024-03-28T15:50:39.336877+0000","last_peered":"2024-03-28T15:50:39.336877+0000","last_clean":"2024-03-28T15:50:39.336877+0000","last_became_active":"2024-03-28T15:50:32.421100+0000","last_became_peered":"2024-03-28T15:50:32.421100+0000","last_unstale":"2024-03-28T15:50:39.336877+0000","last_undegraded":"2024-03-28T15:50:39.336877+0000","last_fullsized":"2024-03-28T15:50:39.336877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:46:20.477403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825710+0000","last_change":"2024-03-28T15:50:36.457721+0000","last_active":"2024-03-28T15:50:37.825710+0000","last_peered":"2024-03-28T15:50:37.825710+0000","last_clean":"2024-03-28T15:50:37.825710+0000","last_became_active":"2024-03-28T15:50:36.457028+0000","last_became_peered":"2024-03-28T15:50:36.457028+0000","last_unstale":"2024-03-28T15:50:37.825710+0000","last_undegraded":"2024-03-28T15:50:37.825710+0000","last_fullsized":"2024-03-28T15:50:37.825710+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:47.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442820+0000","last_change":"2024-03-28T15:50:31.418265+0000","last_active":"2024-03-28T15:50:37.442820+0000","last_peered":"2024-03-28T15:50:37.442820+0000","last_clean":"2024-03-28T15:50:37.442820+0000","last_became_active":"2024-03-28T15:50:31.417944+0000","last_became_peered":"2024-03-28T15:50:31.417944+0000","last_unstale":"2024-03-28T15:50:37.442820+0000","last_undegraded":"2024-03-28T15:50:37.442820+0000","last_fullsized":"2024-03-28T15:50:37.442820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:58.633512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412512+0000","last_change":"2024-03-28T15:50:32.433180+0000","last_active":"2024-03-28T15:50:38.412512+0000","last_peered":"2024-03-28T15:50:38.412512+0000","last_clean":"2024-03-28T15:50:38.412512+0000","last_became_active":"2024-03-28T15:50:32.432598+0000","last_became_peered":"2024-03-28T15:50:32.432598+0000","last_unstale":"2024-03-28T15:50:38.412512+0000","last_undegraded":"2024-03-28T15:50:38.412512+0000","last_fullsized":"2024-03-28T15:50:38.412512+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:16.051480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336275+0000","last_change":"2024-03-28T15:50:36.453471+0000","last_active":"2024-03-28T15:50:39.336275+0000","last_peered":"2024-03-28T15:50:39.336275+0000","last_clean":"2024-03-28T15:50:39.336275+0000","last_became_active":"2024-03-28T15:50:36.453070+0000","last_became_peered":"2024-03-28T15:50:36.453070+0000","last_unstale":"2024-03-28T15:50:39.336275+0000","last_undegraded":"2024-03-28T15:50:39.336275+0000","last_fullsized":"2024-03-28T15:50:39.336275+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:55.417086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"90'21","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:59.597314+0000","last_change":"2024-03-28T15:50:31.419585+0000","last_active":"2024-03-28T15:50:59.597314+0000","last_peered":"2024-03-28T15:50:59.597314+0000","last_clean":"2024-03-28T15:50:59.597314+0000","last_became_active":"2024-03-28T15:50:31.419241+0000","last_became_peered":"2024-03-28T15:50:31.419241+0000","last_unstale":"2024-03-28T15:50:59.597314+0000","last_undegraded":"2024-03-28T15:50:59.597314+0000","last_fullsized":"2024-03-28T15:50:59.597314+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:34:17.438115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23141,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":23,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632228+0000","last_change":"2024-03-28T15:50:32.436526+0000","last_active":"2024-03-28T15:50:38.632228+0000","last_peered":"2024-03-28T15:50:38.632228+0000","last_clean":"2024-03-28T15:50:38.632228+0000","last_became_active":"2024-03-28T15:50:32.436211+0000","last_became_peered":"2024-03-28T15:50:32.436211+0000","last_unstale":"2024-03-28T15:50:38.632228+0000","last_undegraded":"2024-03-28T15:50:38.632228+0000","last_fullsized":"2024-03-28T15:50:38.632228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:04.424281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443204+0000","last_change":"2024-03-28T15:50:36.460308+0000","last_active":"2024-03-28T15:50:37.443204+0000","last_peered":"2024-03-28T15:50:37.443204+0000","last_clean":"2024-03-28T15:50:37.443204+0000","last_became_active":"2024-03-28T15:50:36.459417+0000","last_became_peered":"2024-03-28T15:50:36.459417+0000","last_unstale":"2024-03-28T15:50:37.443204+0000","last_undegraded":"2024-03-28T15:50:37.443204+0000","last_fullsized":"2024-03-28T15:50:37.443204+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:22.022249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'6","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567379+0000","last_change":"2024-03-28T15:50:31.426829+0000","last_active":"2024-03-28T15:50:57.567379+0000","last_peered":"2024-03-28T15:50:57.567379+0000","last_clean":"2024-03-28T15:50:57.567379+0000","last_became_active":"2024-03-28T15:50:31.426434+0000","last_became_peered":"2024-03-28T15:50:31.426434+0000","last_unstale":"2024-03-28T15:50:57.567379+0000","last_undegraded":"2024-03-28T15:50:57.567379+0000","last_fullsized":"2024-03-28T15:50:57.567379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.567766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441951+0000","last_change":"2024-03-28T15:50:32.427927+0000","last_active":"2024-03-28T15:50:37.441951+0000","last_peered":"2024-03-28T15:50:37.441951+0000","last_clean":"2024-03-28T15:50:37.441951+0000","last_became_active":"2024-03-28T15:50:32.427609+0000","last_became_peered":"2024-03-28T15:50:32.427609+0000","last_unstale":"2024-03-28T15:50:37.441951+0000","last_undegraded":"2024-03-28T15:50:37.441951+0000","last_fullsized":"2024-03-28T15:50:37.441951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:49.120638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241396+0000","last_change":"2024-03-28T15:50:36.460355+0000","last_active":"2024-03-28T15:50:39.241396+0000","last_peered":"2024-03-28T15:50:39.241396+0000","last_clean":"2024-03-28T15:50:39.241396+0000","last_became_active":"2024-03-28T15:50:36.459935+0000","last_became_peered":"2024-03-28T15:50:36.459935+0000","last_unstale":"2024-03-28T15:50:39.241396+0000","last_undegraded":"2024-03-28T15:50:39.241396+0000","last_fullsized":"2024-03-28T15:50:39.241396+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.895844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'3","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561849+0000","last_change":"2024-03-28T15:50:31.417282+0000","last_active":"2024-03-28T15:50:56.561849+0000","last_peered":"2024-03-28T15:50:56.561849+0000","last_clean":"2024-03-28T15:50:56.561849+0000","last_became_active":"2024-03-28T15:50:31.416852+0000","last_became_peered":"2024-03-28T15:50:31.416852+0000","last_unstale":"2024-03-28T15:50:56.561849+0000","last_undegraded":"2024-03-28T15:50:56.561849+0000","last_fullsized":"2024-03-28T15:50:56.561849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:26:12.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444022+0000","last_change":"2024-03-28T15:50:32.430362+0000","last_active":"2024-03-28T15:50:37.444022+0000","last_peered":"2024-03-28T15:50:37.444022+0000","last_clean":"2024-03-28T15:50:37.444022+0000","last_became_active":"2024-03-28T15:50:32.429999+0000","last_became_peered":"2024-03-28T15:50:32.429999+0000","last_unstale":"2024-03-28T15:50:37.444022+0000","last_undegraded":"2024-03-28T15:50:37.444022+0000","last_fullsized":"2024-03-28T15:50:37.444022+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:47.613322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336702+0000","last_change":"2024-03-28T15:50:36.463116+0000","last_active":"2024-03-28T15:50:39.336702+0000","last_peered":"2024-03-28T15:50:39.336702+0000","last_clean":"2024-03-28T15:50:39.336702+0000","last_became_active":"2024-03-28T15:50:36.462705+0000","last_became_peered":"2024-03-28T15:50:36.462705+0000","last_unstale":"2024-03-28T15:50:39.336702+0000","last_undegraded":"2024-03-28T15:50:39.336702+0000","last_fullsized":"2024-03-28T15:50:39.336702+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:49.876404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.564767+0000","last_change":"2024-03-28T15:50:31.418595+0000","last_active":"2024-03-28T15:50:57.564767+0000","last_peered":"2024-03-28T15:50:57.564767+0000","last_clean":"2024-03-28T15:50:57.564767+0000","last_became_active":"2024-03-28T15:50:31.418185+0000","last_became_peered":"2024-03-28T15:50:31.418185+0000","last_unstale":"2024-03-28T15:50:57.564767+0000","last_undegraded":"2024-03-28T15:50:57.564767+0000","last_fullsized":"2024-03-28T15:50:57.564767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:40.246760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443807+0000","last_change":"2024-03-28T15:50:32.431867+0000","last_active":"2024-03-28T15:50:37.443807+0000","last_peered":"2024-03-28T15:50:37.443807+0000","last_clean":"2024-03-28T15:50:37.443807+0000","last_became_active":"2024-03-28T15:50:32.431552+0000","last_became_peered":"2024-03-28T15:50:32.431552+0000","last_unstale":"2024-03-28T15:50:37.443807+0000","last_undegraded":"2024-03-28T15:50:37.443807+0000","last_fullsized":"2024-03-28T15:50:37.443807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:20.849087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442219+0000","last_change":"2024-03-28T15:50:36.466785+0000","last_active":"2024-03-28T15:50:37.442219+0000","last_peered":"2024-03-28T15:50:37.442219+0000","last_clean":"2024-03-28T15:50:37.442219+0000","last_became_active":"2024-03-28T15:50:36.466356+0000","last_became_peered":"2024-03-28T15:50:36.466356+0000","last_unstale":"2024-03-28T15:50:37.442219+0000","last_undegraded":"2024-03-28T15:50:37.442219+0000","last_fullsized":"2024-03-28T15:50:37.442219+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:07.453802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'10","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561652+0000","last_change":"2024-03-28T15:50:31.417956+0000","last_active":"2024-03-28T15:50:56.561652+0000","last_peered":"2024-03-28T15:50:56.561652+0000","last_clean":"2024-03-28T15:50:56.561652+0000","last_became_active":"2024-03-28T15:50:31.417613+0000","last_became_peered":"2024-03-28T15:50:31.417613+0000","last_unstale":"2024-03-28T15:50:56.561652+0000","last_undegraded":"2024-03-28T15:50:56.561652+0000","last_fullsized":"2024-03-28T15:50:56.561652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:33.039685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442257+0000","last_change":"2024-03-28T15:50:32.426973+0000","last_active":"2024-03-28T15:50:37.442257+0000","last_peered":"2024-03-28T15:50:37.442257+0000","last_clean":"2024-03-28T15:50:37.442257+0000","last_became_active":"2024-03-28T15:50:32.426668+0000","last_became_peered":"2024-03-28T15:50:32.426668+0000","last_unstale":"2024-03-28T15:50:37.442257+0000","last_undegraded":"2024-03-28T15:50:37.442257+0000","last_fullsized":"2024-03-28T15:50:37.442257+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:10:56.270066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442247+0000","last_change":"2024-03-28T15:50:36.463509+0000","last_active":"2024-03-28T15:50:37.442247+0000","last_peered":"2024-03-28T15:50:37.442247+0000","last_clean":"2024-03-28T15:50:37.442247+0000","last_became_active":"2024-03-28T15:50:36.462733+0000","last_became_peered":"2024-03-28T15:50:36.462733+0000","last_unstale":"2024-03-28T15:50:37.442247+0000","last_undegraded":"2024-03-28T15:50:37.442247+0000","last_fullsized":"2024-03-28T15:50:37.442247+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:43:33.737012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632394+0000","last_change":"2024-03-28T15:50:31.420224+0000","last_active":"2024-03-28T15:50:38.632394+0000","last_peered":"2024-03-28T15:50:38.632394+0000","last_clean":"2024-03-28T15:50:38.632394+0000","last_became_active":"2024-03-28T15:50:31.419913+0000","last_became_peered":"2024-03-28T15:50:31.419913+0000","last_unstale":"2024-03-28T15:50:38.632394+0000","last_undegraded":"2024-03-28T15:50:38.632394+0000","last_fullsized":"2024-03-28T15:50:38.632394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:59:49.834297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241516+0000","last_change":"2024-03-28T15:50:32.425419+0000","last_active":"2024-03-28T15:50:39.241516+0000","last_peered":"2024-03-28T15:50:39.241516+0000","last_clean":"2024-03-28T15:50:39.241516+0000","last_became_active":"2024-03-28T15:50:32.425129+0000","last_became_peered":"2024-03-28T15:50:32.425129+0000","last_unstale":"2024-03-28T15:50:39.241516+0000","last_undegraded":"2024-03-28T15:50:39.241516+0000","last_fullsized":"2024-03-28T15:50:39.241516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:35:32.407032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632095+0000","last_change":"2024-03-28T15:50:36.460744+0000","last_active":"2024-03-28T15:50:38.632095+0000","last_peered":"2024-03-28T15:50:38.632095+0000","last_clean":"2024-03-28T15:50:38.632095+0000","last_became_active":"2024-03-28T15:50:36.460276+0000","last_became_peered":"2024-03-28T15:50:36.460276+0000","last_unstale":"2024-03-28T15:50:38.632095+0000","last_undegraded":"2024-03-28T15:50:38.632095+0000","last_fullsized":"2024-03-28T15:50:38.632095+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:12.600154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336564+0000","last_change":"2024-03-28T15:50:31.427718+0000","last_active":"2024-03-28T15:50:39.336564+0000","last_peered":"2024-03-28T15:50:39.336564+0000","last_clean":"2024-03-28T15:50:39.336564+0000","last_became_active":"2024-03-28T15:50:31.427414+0000","last_became_peered":"2024-03-28T15:50:31.427414+0000","last_unstale":"2024-03-28T15:50:39.336564+0000","last_undegraded":"2024-03-28T15:50:39.336564+0000","last_fullsized":"2024-03-28T15:50:39.336564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:24.636431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336159+0000","last_change":"2024-03-28T15:50:32.424409+0000","last_active":"2024-03-28T15:50:39.336159+0000","last_peered":"2024-03-28T15:50:39.336159+0000","last_clean":"2024-03-28T15:50:39.336159+0000","last_became_active":"2024-03-28T15:50:32.424108+0000","last_became_peered":"2024-03-28T15:50:32.424108+0000","last_unstale":"2024-03-28T15:50:39.336159+0000","last_undegraded":"2024-03-28T15:50:39.336159+0000","last_fullsized":"2024-03-28T15:50:39.336159+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:59:23.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442262+0000","last_change":"2024-03-28T15:50:36.463533+0000","last_active":"2024-03-28T15:50:37.442262+0000","last_peered":"2024-03-28T15:50:37.442262+0000","last_clean":"2024-03-28T15:50:37.442262+0000","last_became_active":"2024-03-28T15:50:36.463047+0000","last_became_peered":"2024-03-28T15:50:36.463047+0000","last_unstale":"2024-03-28T15:50:37.442262+0000","last_undegraded":"2024-03-28T15:50:37.442262+0000","last_fullsized":"2024-03-28T15:50:37.442262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:52.072250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443761+0000","last_change":"2024-03-28T15:50:31.417073+0000","last_active":"2024-03-28T15:50:37.443761+0000","last_peered":"2024-03-28T15:50:37.443761+0000","last_clean":"2024-03-28T15:50:37.443761+0000","last_became_active":"2024-03-28T15:50:31.416755+0000","last_became_peered":"2024-03-28T15:50:31.416755+0000","last_unstale":"2024-03-28T15:50:37.443761+0000","last_undegraded":"2024-03-28T15:50:37.443761+0000","last_fullsized":"2024-03-28T15:50:37.443761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:45.887894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336211+0000","last_change":"2024-03-28T15:50:32.424912+0000","last_active":"2024-03-28T15:50:39.336211+0000","last_peered":"2024-03-28T15:50:39.336211+0000","last_clean":"2024-03-28T15:50:39.336211+0000","last_became_active":"2024-03-28T15:50:32.424609+0000","last_became_peered":"2024-03-28T15:50:32.424609+0000","last_unstale":"2024-03-28T15:50:39.336211+0000","last_undegraded":"2024-03-28T15:50:39.336211+0000","last_fullsized":"2024-03-28T15:50:39.336211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:37.539793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441937+0000","last_change":"2024-03-28T15:50:36.463968+0000","last_active":"2024-03-28T15:50:37.441937+0000","last_peered":"2024-03-28T15:50:37.441937+0000","last_clean":"2024-03-28T15:50:37.441937+0000","last_became_active":"2024-03-28T15:50:36.460697+0000","last_became_peered":"2024-03-28T15:50:36.460697+0000","last_unstale":"2024-03-28T15:50:37.441937+0000","last_undegraded":"2024-03-28T15:50:37.441937+0000","last_fullsized":"2024-03-28T15:50:37.441937+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:00.726769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442247+0000","last_change":"2024-03-28T15:50:32.419783+0000","last_active":"2024-03-28T15:50:37.442247+0000","last_peered":"2024-03-28T15:50:37.442247+0000","last_clean":"2024-03-28T15:50:37.442247+0000","last_became_active":"2024-03-28T15:50:32.419404+0000","last_became_peered":"2024-03-28T15:50:32.419404+0000","last_unstale":"2024-03-28T15:50:37.442247+0000","last_undegraded":"2024-03-28T15:50:37.442247+0000","last_fullsized":"2024-03-28T15:50:37.442247+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:42.437830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.568781+0000","last_change":"2024-03-28T15:50:31.414630+0000","last_active":"2024-03-28T15:50:57.568781+0000","last_peered":"2024-03-28T15:50:57.568781+0000","last_clean":"2024-03-28T15:50:57.568781+0000","last_became_active":"2024-03-28T15:50:31.414300+0000","last_became_peered":"2024-03-28T15:50:31.414300+0000","last_unstale":"2024-03-28T15:50:57.568781+0000","last_undegraded":"2024-03-28T15:50:57.568781+0000","last_fullsized":"2024-03-28T15:50:57.568781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:48.773649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443984+0000","last_change":"2024-03-28T15:50:36.462352+0000","last_active":"2024-03-28T15:50:37.443984+0000","last_peered":"2024-03-28T15:50:37.443984+0000","last_clean":"2024-03-28T15:50:37.443984+0000","last_became_active":"2024-03-28T15:50:36.462003+0000","last_became_peered":"2024-03-28T15:50:36.462003+0000","last_unstale":"2024-03-28T15:50:37.443984+0000","last_undegraded":"2024-03-28T15:50:37.443984+0000","last_fullsized":"2024-03-28T15:50:37.443984+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:42.830355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343849+0000","last_change":"2024-03-28T15:50:32.429086+0000","last_active":"2024-03-28T15:50:38.343849+0000","last_peered":"2024-03-28T15:50:38.343849+0000","last_clean":"2024-03-28T15:50:38.343849+0000","last_became_active":"2024-03-28T15:50:32.427761+0000","last_became_peered":"2024-03-28T15:50:32.427761+0000","last_unstale":"2024-03-28T15:50:38.343849+0000","last_undegraded":"2024-03-28T15:50:38.343849+0000","last_fullsized":"2024-03-28T15:50:38.343849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.883733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561837+0000","last_change":"2024-03-28T15:50:31.417017+0000","last_active":"2024-03-28T15:50:56.561837+0000","last_peered":"2024-03-28T15:50:56.561837+0000","last_clean":"2024-03-28T15:50:56.561837+0000","last_became_active":"2024-03-28T15:50:31.416688+0000","last_became_peered":"2024-03-28T15:50:31.416688+0000","last_unstale":"2024-03-28T15:50:56.561837+0000","last_undegraded":"2024-03-28T15:50:56.561837+0000","last_fullsized":"2024-03-28T15:50:56.561837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:39.364544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":355,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443656+0000","last_change":"2024-03-28T15:49:22.209051+0000","last_active":"2024-03-28T15:50:37.443656+0000","last_peered":"2024-03-28T15:50:37.443656+0000","last_clean":"2024-03-28T15:50:37.443656+0000","last_became_active":"2024-03-28T15:49:22.208633+0000","last_became_peered":"2024-03-28T15:49:22.208633+0000","last_unstale":"2024-03-28T15:50:37.443656+0000","last_undegraded":"2024-03-28T15:50:37.443656+0000","last_fullsized":"2024-03-28T15:50:37.443656+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_clean_scrub_stamp":"2024-03-28T15:45:59.266253+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:08:41.194614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442933+0000","last_change":"2024-03-28T15:50:36.463514+0000","last_active":"2024-03-28T15:50:37.442933+0000","last_peered":"2024-03-28T15:50:37.442933+0000","last_clean":"2024-03-28T15:50:37.442933+0000","last_became_active":"2024-03-28T15:50:36.463110+0000","last_became_peered":"2024-03-28T15:50:36.463110+0000","last_unstale":"2024-03-28T15:50:37.442933+0000","last_undegraded":"2024-03-28T15:50:37.442933+0000","last_fullsized":"2024-03-28T15:50:37.442933+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:30.481123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441412+0000","last_change":"2024-03-28T15:50:32.427016+0000","last_active":"2024-03-28T15:50:37.441412+0000","last_peered":"2024-03-28T15:50:37.441412+0000","last_clean":"2024-03-28T15:50:37.441412+0000","last_became_active":"2024-03-28T15:50:32.426519+0000","last_became_peered":"2024-03-28T15:50:32.426519+0000","last_unstale":"2024-03-28T15:50:37.441412+0000","last_undegraded":"2024-03-28T15:50:37.441412+0000","last_fullsized":"2024-03-28T15:50:37.441412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:27.650290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443828+0000","last_change":"2024-03-28T15:50:31.420871+0000","last_active":"2024-03-28T15:50:37.443828+0000","last_peered":"2024-03-28T15:50:37.443828+0000","last_clean":"2024-03-28T15:50:37.443828+0000","last_became_active":"2024-03-28T15:50:31.420418+0000","last_became_peered":"2024-03-28T15:50:31.420418+0000","last_unstale":"2024-03-28T15:50:37.443828+0000","last_undegraded":"2024-03-28T15:50:37.443828+0000","last_fullsized":"2024-03-28T15:50:37.443828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:57.615539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241505+0000","last_change":"2024-03-28T15:50:36.457759+0000","last_active":"2024-03-28T15:50:39.241505+0000","last_peered":"2024-03-28T15:50:39.241505+0000","last_clean":"2024-03-28T15:50:39.241505+0000","last_became_active":"2024-03-28T15:50:36.457248+0000","last_became_peered":"2024-03-28T15:50:36.457248+0000","last_unstale":"2024-03-28T15:50:39.241505+0000","last_undegraded":"2024-03-28T15:50:39.241505+0000","last_fullsized":"2024-03-28T15:50:39.241505+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:13:30.420180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.632303+0000","last_change":"2024-03-28T15:50:31.428270+0000","last_active":"2024-03-28T15:50:38.632303+0000","last_peered":"2024-03-28T15:50:38.632303+0000","last_clean":"2024-03-28T15:50:38.632303+0000","last_became_active":"2024-03-28T15:50:31.427888+0000","last_became_peered":"2024-03-28T15:50:31.427888+0000","last_unstale":"2024-03-28T15:50:38.632303+0000","last_undegraded":"2024-03-28T15:50:38.632303+0000","last_fullsized":"2024-03-28T15:50:38.632303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:48:51.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442183+0000","last_change":"2024-03-28T15:50:32.416631+0000","last_active":"2024-03-28T15:50:37.442183+0000","last_peered":"2024-03-28T15:50:37.442183+0000","last_clean":"2024-03-28T15:50:37.442183+0000","last_became_active":"2024-03-28T15:50:32.416300+0000","last_became_peered":"2024-03-28T15:50:32.416300+0000","last_unstale":"2024-03-28T15:50:37.442183+0000","last_undegraded":"2024-03-28T15:50:37.442183+0000","last_fullsized":"2024-03-28T15:50:37.442183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:19.158562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343845+0000","last_change":"2024-03-28T15:50:36.465991+0000","last_active":"2024-03-28T15:50:38.343845+0000","last_peered":"2024-03-28T15:50:38.343845+0000","last_clean":"2024-03-28T15:50:38.343845+0000","last_became_active":"2024-03-28T15:50:36.465506+0000","last_became_peered":"2024-03-28T15:50:36.465506+0000","last_unstale":"2024-03-28T15:50:38.343845+0000","last_undegraded":"2024-03-28T15:50:38.343845+0000","last_fullsized":"2024-03-28T15:50:38.343845+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:22.889068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"90'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.575837+0000","last_change":"2024-03-28T15:50:31.423894+0000","last_active":"2024-03-28T15:50:58.575837+0000","last_peered":"2024-03-28T15:50:58.575837+0000","last_clean":"2024-03-28T15:50:58.575837+0000","last_became_active":"2024-03-28T15:50:31.423562+0000","last_became_peered":"2024-03-28T15:50:31.423562+0000","last_unstale":"2024-03-28T15:50:58.575837+0000","last_undegraded":"2024-03-28T15:50:58.575837+0000","last_fullsized":"2024-03-28T15:50:58.575837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:55.065632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343744+0000","last_change":"2024-03-28T15:50:32.435132+0000","last_active":"2024-03-28T15:50:38.343744+0000","last_peered":"2024-03-28T15:50:38.343744+0000","last_clean":"2024-03-28T15:50:38.343744+0000","last_became_active":"2024-03-28T15:50:32.434523+0000","last_became_peered":"2024-03-28T15:50:32.434523+0000","last_unstale":"2024-03-28T15:50:38.343744+0000","last_undegraded":"2024-03-28T15:50:38.343744+0000","last_fullsized":"2024-03-28T15:50:38.343744+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:35.363388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336027+0000","last_change":"2024-03-28T15:50:36.453501+0000","last_active":"2024-03-28T15:50:39.336027+0000","last_peered":"2024-03-28T15:50:39.336027+0000","last_clean":"2024-03-28T15:50:39.336027+0000","last_became_active":"2024-03-28T15:50:36.452121+0000","last_became_peered":"2024-03-28T15:50:36.452121+0000","last_unstale":"2024-03-28T15:50:39.336027+0000","last_undegraded":"2024-03-28T15:50:39.336027+0000","last_fullsized":"2024-03-28T15:50:39.336027+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:48:33.354233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.560926+0000","last_change":"2024-03-28T15:50:31.425645+0000","last_active":"2024-03-28T15:50:56.560926+0000","last_peered":"2024-03-28T15:50:56.560926+0000","last_clean":"2024-03-28T15:50:56.560926+0000","last_became_active":"2024-03-28T15:50:31.425332+0000","last_became_peered":"2024-03-28T15:50:31.425332+0000","last_unstale":"2024-03-28T15:50:56.560926+0000","last_undegraded":"2024-03-28T15:50:56.560926+0000","last_fullsized":"2024-03-28T15:50:56.560926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:16.221936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336521+0000","last_change":"2024-03-28T15:50:32.432602+0000","last_active":"2024-03-28T15:50:39.336521+0000","last_peered":"2024-03-28T15:50:39.336521+0000","last_clean":"2024-03-28T15:50:39.336521+0000","last_became_active":"2024-03-28T15:50:32.432303+0000","last_became_peered":"2024-03-28T15:50:32.432303+0000","last_unstale":"2024-03-28T15:50:39.336521+0000","last_undegraded":"2024-03-28T15:50:39.336521+0000","last_fullsized":"2024-03-28T15:50:39.336521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.524582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825616+0000","last_change":"2024-03-28T15:50:36.457991+0000","last_active":"2024-03-28T15:50:37.825616+0000","last_peered":"2024-03-28T15:50:37.825616+0000","last_clean":"2024-03-28T15:50:37.825616+0000","last_became_active":"2024-03-28T15:50:36.457380+0000","last_became_peered":"2024-03-28T15:50:36.457380+0000","last_unstale":"2024-03-28T15:50:37.825616+0000","last_undegraded":"2024-03-28T15:50:37.825616+0000","last_fullsized":"2024-03-28T15:50:37.825616+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:28:19.866888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.559756+0000","last_change":"2024-03-28T15:50:31.418403+0000","last_active":"2024-03-28T15:50:56.559756+0000","last_peered":"2024-03-28T15:50:56.559756+0000","last_clean":"2024-03-28T15:50:56.559756+0000","last_became_active":"2024-03-28T15:50:31.418102+0000","last_became_peered":"2024-03-28T15:50:31.418102+0000","last_unstale":"2024-03-28T15:50:56.559756+0000","last_undegraded":"2024-03-28T15:50:56.559756+0000","last_fullsized":"2024-03-28T15:50:56.559756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:01:56.175404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.631808+0000","last_change":"2024-03-28T15:50:32.417331+0000","last_active":"2024-03-28T15:50:38.631808+0000","last_peered":"2024-03-28T15:50:38.631808+0000","last_clean":"2024-03-28T15:50:38.631808+0000","last_became_active":"2024-03-28T15:50:32.417026+0000","last_became_peered":"2024-03-28T15:50:32.417026+0000","last_unstale":"2024-03-28T15:50:38.631808+0000","last_undegraded":"2024-03-28T15:50:38.631808+0000","last_fullsized":"2024-03-28T15:50:38.631808+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:24.081575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336959+0000","last_change":"2024-03-28T15:50:36.454877+0000","last_active":"2024-03-28T15:50:39.336959+0000","last_peered":"2024-03-28T15:50:39.336959+0000","last_clean":"2024-03-28T15:50:39.336959+0000","last_became_active":"2024-03-28T15:50:36.453802+0000","last_became_peered":"2024-03-28T15:50:36.453802+0000","last_unstale":"2024-03-28T15:50:39.336959+0000","last_undegraded":"2024-03-28T15:50:39.336959+0000","last_fullsized":"2024-03-28T15:50:39.336959+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:56.479558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'5","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:58.575409+0000","last_change":"2024-03-28T15:50:31.425672+0000","last_active":"2024-03-28T15:50:58.575409+0000","last_peered":"2024-03-28T15:50:58.575409+0000","last_clean":"2024-03-28T15:50:58.575409+0000","last_became_active":"2024-03-28T15:50:31.425324+0000","last_became_peered":"2024-03-28T15:50:31.425324+0000","last_unstale":"2024-03-28T15:50:58.575409+0000","last_undegraded":"2024-03-28T15:50:58.575409+0000","last_fullsized":"2024-03-28T15:50:58.575409+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:48:46.380962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241161+0000","last_change":"2024-03-28T15:50:32.424831+0000","last_active":"2024-03-28T15:50:39.241161+0000","last_peered":"2024-03-28T15:50:39.241161+0000","last_clean":"2024-03-28T15:50:39.241161+0000","last_became_active":"2024-03-28T15:50:32.423678+0000","last_became_peered":"2024-03-28T15:50:32.423678+0000","last_unstale":"2024-03-28T15:50:39.241161+0000","last_undegraded":"2024-03-28T15:50:39.241161+0000","last_fullsized":"2024-03-28T15:50:39.241161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:35:06.895634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825514+0000","last_change":"2024-03-28T15:50:36.460255+0000","last_active":"2024-03-28T15:50:37.825514+0000","last_peered":"2024-03-28T15:50:37.825514+0000","last_clean":"2024-03-28T15:50:37.825514+0000","last_became_active":"2024-03-28T15:50:36.459832+0000","last_became_peered":"2024-03-28T15:50:36.459832+0000","last_unstale":"2024-03-28T15:50:37.825514+0000","last_undegraded":"2024-03-28T15:50:37.825514+0000","last_fullsized":"2024-03-28T15:50:37.825514+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:13.911621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'1","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336045+0000","last_change":"2024-03-28T15:50:31.430248+0000","last_active":"2024-03-28T15:50:39.336045+0000","last_peered":"2024-03-28T15:50:39.336045+0000","last_clean":"2024-03-28T15:50:39.336045+0000","last_became_active":"2024-03-28T15:50:31.429964+0000","last_became_peered":"2024-03-28T15:50:31.429964+0000","last_unstale":"2024-03-28T15:50:39.336045+0000","last_undegraded":"2024-03-28T15:50:39.336045+0000","last_fullsized":"2024-03-28T15:50:39.336045+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:10:25.193102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343825+0000","last_change":"2024-03-28T15:50:32.431687+0000","last_active":"2024-03-28T15:50:38.343825+0000","last_peered":"2024-03-28T15:50:38.343825+0000","last_clean":"2024-03-28T15:50:38.343825+0000","last_became_active":"2024-03-28T15:50:32.431295+0000","last_became_peered":"2024-03-28T15:50:32.431295+0000","last_unstale":"2024-03-28T15:50:38.343825+0000","last_undegraded":"2024-03-28T15:50:38.343825+0000","last_fullsized":"2024-03-28T15:50:38.343825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:46:12.055577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.825611+0000","last_change":"2024-03-28T15:50:36.459198+0000","last_active":"2024-03-28T15:50:37.825611+0000","last_peered":"2024-03-28T15:50:37.825611+0000","last_clean":"2024-03-28T15:50:37.825611+0000","last_became_active":"2024-03-28T15:50:36.458815+0000","last_became_peered":"2024-03-28T15:50:36.458815+0000","last_unstale":"2024-03-28T15:50:37.825611+0000","last_undegraded":"2024-03-28T15:50:37.825611+0000","last_fullsized":"2024-03-28T15:50:37.825611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:19.302434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:56.561912+0000","last_change":"2024-03-28T15:50:31.421319+0000","last_active":"2024-03-28T15:50:56.561912+0000","last_peered":"2024-03-28T15:50:56.561912+0000","last_clean":"2024-03-28T15:50:56.561912+0000","last_became_active":"2024-03-28T15:50:31.420399+0000","last_became_peered":"2024-03-28T15:50:31.420399+0000","last_unstale":"2024-03-28T15:50:56.561912+0000","last_undegraded":"2024-03-28T15:50:56.561912+0000","last_fullsized":"2024-03-28T15:50:56.561912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:45.594801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.443793+0000","last_change":"2024-03-28T15:50:32.436155+0000","last_active":"2024-03-28T15:50:37.443793+0000","last_peered":"2024-03-28T15:50:37.443793+0000","last_clean":"2024-03-28T15:50:37.443793+0000","last_became_active":"2024-03-28T15:50:32.435453+0000","last_became_peered":"2024-03-28T15:50:32.435453+0000","last_unstale":"2024-03-28T15:50:37.443793+0000","last_undegraded":"2024-03-28T15:50:37.443793+0000","last_fullsized":"2024-03-28T15:50:37.443793+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:44.159744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412308+0000","last_change":"2024-03-28T15:50:36.456103+0000","last_active":"2024-03-28T15:50:38.412308+0000","last_peered":"2024-03-28T15:50:38.412308+0000","last_clean":"2024-03-28T15:50:38.412308+0000","last_became_active":"2024-03-28T15:50:36.455724+0000","last_became_peered":"2024-03-28T15:50:36.455724+0000","last_unstale":"2024-03-28T15:50:38.412308+0000","last_undegraded":"2024-03-28T15:50:38.412308+0000","last_fullsized":"2024-03-28T15:50:38.412308+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:30.874245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'3","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:57.567315+0000","last_change":"2024-03-28T15:50:31.423648+0000","last_active":"2024-03-28T15:50:57.567315+0000","last_peered":"2024-03-28T15:50:57.567315+0000","last_clean":"2024-03-28T15:50:57.567315+0000","last_became_active":"2024-03-28T15:50:31.423343+0000","last_became_peered":"2024-03-28T15:50:31.423343+0000","last_unstale":"2024-03-28T15:50:57.567315+0000","last_undegraded":"2024-03-28T15:50:57.567315+0000","last_fullsized":"2024-03-28T15:50:57.567315+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:22.475804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444118+0000","last_change":"2024-03-28T15:50:32.432943+0000","last_active":"2024-03-28T15:50:37.444118+0000","last_peered":"2024-03-28T15:50:37.444118+0000","last_clean":"2024-03-28T15:50:37.444118+0000","last_became_active":"2024-03-28T15:50:32.432535+0000","last_became_peered":"2024-03-28T15:50:32.432535+0000","last_unstale":"2024-03-28T15:50:37.444118+0000","last_undegraded":"2024-03-28T15:50:37.444118+0000","last_fullsized":"2024-03-28T15:50:37.444118+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:20:17.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336101+0000","last_change":"2024-03-28T15:50:36.463152+0000","last_active":"2024-03-28T15:50:39.336101+0000","last_peered":"2024-03-28T15:50:39.336101+0000","last_clean":"2024-03-28T15:50:39.336101+0000","last_became_active":"2024-03-28T15:50:36.462771+0000","last_became_peered":"2024-03-28T15:50:36.462771+0000","last_unstale":"2024-03-28T15:50:39.336101+0000","last_undegraded":"2024-03-28T15:50:39.336101+0000","last_fullsized":"2024-03-28T15:50:39.336101+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:09:22.958078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":23,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442903+0000","last_change":"2024-03-28T15:50:31.419043+0000","last_active":"2024-03-28T15:50:37.442903+0000","last_peered":"2024-03-28T15:50:37.442903+0000","last_clean":"2024-03-28T15:50:37.442903+0000","last_became_active":"2024-03-28T15:50:31.418722+0000","last_became_peered":"2024-03-28T15:50:31.418722+0000","last_unstale":"2024-03-28T15:50:37.442903+0000","last_undegraded":"2024-03-28T15:50:37.442903+0000","last_fullsized":"2024-03-28T15:50:37.442903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:10.512442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444051+0000","last_change":"2024-03-28T15:50:32.431708+0000","last_active":"2024-03-28T15:50:37.444051+0000","last_peered":"2024-03-28T15:50:37.444051+0000","last_clean":"2024-03-28T15:50:37.444051+0000","last_became_active":"2024-03-28T15:50:32.431356+0000","last_became_peered":"2024-03-28T15:50:32.431356+0000","last_unstale":"2024-03-28T15:50:37.444051+0000","last_undegraded":"2024-03-28T15:50:37.444051+0000","last_fullsized":"2024-03-28T15:50:37.444051+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:23:49.409548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442042+0000","last_change":"2024-03-28T15:50:36.455924+0000","last_active":"2024-03-28T15:50:37.442042+0000","last_peered":"2024-03-28T15:50:37.442042+0000","last_clean":"2024-03-28T15:50:37.442042+0000","last_became_active":"2024-03-28T15:50:36.455511+0000","last_became_peered":"2024-03-28T15:50:36.455511+0000","last_unstale":"2024-03-28T15:50:37.442042+0000","last_undegraded":"2024-03-28T15:50:37.442042+0000","last_fullsized":"2024-03-28T15:50:37.442042+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:48.039313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442073+0000","last_change":"2024-03-28T15:50:31.418393+0000","last_active":"2024-03-28T15:50:37.442073+0000","last_peered":"2024-03-28T15:50:37.442073+0000","last_clean":"2024-03-28T15:50:37.442073+0000","last_became_active":"2024-03-28T15:50:31.418086+0000","last_became_peered":"2024-03-28T15:50:31.418086+0000","last_unstale":"2024-03-28T15:50:37.442073+0000","last_undegraded":"2024-03-28T15:50:37.442073+0000","last_fullsized":"2024-03-28T15:50:37.442073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:44.882002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.444185+0000","last_change":"2024-03-28T15:50:32.429008+0000","last_active":"2024-03-28T15:50:37.444185+0000","last_peered":"2024-03-28T15:50:37.444185+0000","last_clean":"2024-03-28T15:50:37.444185+0000","last_became_active":"2024-03-28T15:50:32.428398+0000","last_became_peered":"2024-03-28T15:50:32.428398+0000","last_unstale":"2024-03-28T15:50:37.444185+0000","last_undegraded":"2024-03-28T15:50:37.444185+0000","last_fullsized":"2024-03-28T15:50:37.444185+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:50.908576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241219+0000","last_change":"2024-03-28T15:50:32.424785+0000","last_active":"2024-03-28T15:50:39.241219+0000","last_peered":"2024-03-28T15:50:39.241219+0000","last_clean":"2024-03-28T15:50:39.241219+0000","last_became_active":"2024-03-28T15:50:32.424134+0000","last_became_peered":"2024-03-28T15:50:32.424134+0000","last_unstale":"2024-03-28T15:50:39.241219+0000","last_undegraded":"2024-03-28T15:50:39.241219+0000","last_fullsized":"2024-03-28T15:50:39.241219+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:20.842423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336299+0000","last_change":"2024-03-28T15:50:32.426830+0000","last_active":"2024-03-28T15:50:39.336299+0000","last_peered":"2024-03-28T15:50:39.336299+0000","last_clean":"2024-03-28T15:50:39.336299+0000","last_became_active":"2024-03-28T15:50:32.426492+0000","last_became_peered":"2024-03-28T15:50:32.426492+0000","last_unstale":"2024-03-28T15:50:39.336299+0000","last_undegraded":"2024-03-28T15:50:39.336299+0000","last_fullsized":"2024-03-28T15:50:39.336299+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:50:11.847280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.335941+0000","last_change":"2024-03-28T15:50:32.425448+0000","last_active":"2024-03-28T15:50:39.335941+0000","last_peered":"2024-03-28T15:50:39.335941+0000","last_clean":"2024-03-28T15:50:39.335941+0000","last_became_active":"2024-03-28T15:50:32.425148+0000","last_became_peered":"2024-03-28T15:50:32.425148+0000","last_unstale":"2024-03-28T15:50:39.335941+0000","last_undegraded":"2024-03-28T15:50:39.335941+0000","last_fullsized":"2024-03-28T15:50:39.335941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:31.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336292+0000","last_change":"2024-03-28T15:50:32.433610+0000","last_active":"2024-03-28T15:50:39.336292+0000","last_peered":"2024-03-28T15:50:39.336292+0000","last_clean":"2024-03-28T15:50:39.336292+0000","last_became_active":"2024-03-28T15:50:32.433096+0000","last_became_peered":"2024-03-28T15:50:32.433096+0000","last_unstale":"2024-03-28T15:50:39.336292+0000","last_undegraded":"2024-03-28T15:50:39.336292+0000","last_fullsized":"2024-03-28T15:50:39.336292+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:20.479797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442061+0000","last_change":"2024-03-28T15:50:32.428195+0000","last_active":"2024-03-28T15:50:37.442061+0000","last_peered":"2024-03-28T15:50:37.442061+0000","last_clean":"2024-03-28T15:50:37.442061+0000","last_became_active":"2024-03-28T15:50:32.427877+0000","last_became_peered":"2024-03-28T15:50:32.427877+0000","last_unstale":"2024-03-28T15:50:37.442061+0000","last_undegraded":"2024-03-28T15:50:37.442061+0000","last_fullsized":"2024-03-28T15:50:37.442061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:56.998688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241597+0000","last_change":"2024-03-28T15:50:32.420177+0000","last_active":"2024-03-28T15:50:39.241597+0000","last_peered":"2024-03-28T15:50:39.241597+0000","last_clean":"2024-03-28T15:50:39.241597+0000","last_became_active":"2024-03-28T15:50:32.419805+0000","last_became_peered":"2024-03-28T15:50:32.419805+0000","last_unstale":"2024-03-28T15:50:39.241597+0000","last_undegraded":"2024-03-28T15:50:39.241597+0000","last_fullsized":"2024-03-28T15:50:39.241597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:04:31.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.336588+0000","last_change":"2024-03-28T15:50:32.417500+0000","last_active":"2024-03-28T15:50:39.336588+0000","last_peered":"2024-03-28T15:50:39.336588+0000","last_clean":"2024-03-28T15:50:39.336588+0000","last_became_active":"2024-03-28T15:50:32.416843+0000","last_became_peered":"2024-03-28T15:50:32.416843+0000","last_unstale":"2024-03-28T15:50:39.336588+0000","last_undegraded":"2024-03-28T15:50:39.336588+0000","last_fullsized":"2024-03-28T15:50:39.336588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:18:03.656816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.343844+0000","last_change":"2024-03-28T15:50:32.428655+0000","last_active":"2024-03-28T15:50:38.343844+0000","last_peered":"2024-03-28T15:50:38.343844+0000","last_clean":"2024-03-28T15:50:38.343844+0000","last_became_active":"2024-03-28T15:50:32.428297+0000","last_became_peered":"2024-03-28T15:50:32.428297+0000","last_unstale":"2024-03-28T15:50:38.343844+0000","last_undegraded":"2024-03-28T15:50:38.343844+0000","last_fullsized":"2024-03-28T15:50:38.343844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:55:32.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.442186+0000","last_change":"2024-03-28T15:50:32.422012+0000","last_active":"2024-03-28T15:50:37.442186+0000","last_peered":"2024-03-28T15:50:37.442186+0000","last_clean":"2024-03-28T15:50:37.442186+0000","last_became_active":"2024-03-28T15:50:32.421465+0000","last_became_peered":"2024-03-28T15:50:32.421465+0000","last_unstale":"2024-03-28T15:50:37.442186+0000","last_undegraded":"2024-03-28T15:50:37.442186+0000","last_fullsized":"2024-03-28T15:50:37.442186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:35.235917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.344169+0000","last_change":"2024-03-28T15:50:32.426624+0000","last_active":"2024-03-28T15:50:38.344169+0000","last_peered":"2024-03-28T15:50:38.344169+0000","last_clean":"2024-03-28T15:50:38.344169+0000","last_became_active":"2024-03-28T15:50:32.425307+0000","last_became_peered":"2024-03-28T15:50:32.425307+0000","last_unstale":"2024-03-28T15:50:38.344169+0000","last_undegraded":"2024-03-28T15:50:38.344169+0000","last_fullsized":"2024-03-28T15:50:38.344169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:05.332057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:38.412474+0000","last_change":"2024-03-28T15:50:32.430274+0000","last_active":"2024-03-28T15:50:38.412474+0000","last_peered":"2024-03-28T15:50:38.412474+0000","last_clean":"2024-03-28T15:50:38.412474+0000","last_became_active":"2024-03-28T15:50:32.429147+0000","last_became_peered":"2024-03-28T15:50:32.429147+0000","last_unstale":"2024-03-28T15:50:38.412474+0000","last_undegraded":"2024-03-28T15:50:38.412474+0000","last_fullsized":"2024-03-28T15:50:38.412474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:19.135249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:37.441703+0000","last_change":"2024-03-28T15:50:32.434933+0000","last_active":"2024-03-28T15:50:37.441703+0000","last_peered":"2024-03-28T15:50:37.441703+0000","last_clean":"2024-03-28T15:50:37.441703+0000","last_became_active":"2024-03-28T15:50:32.434612+0000","last_became_peered":"2024-03-28T15:50:32.434612+0000","last_unstale":"2024-03-28T15:50:37.441703+0000","last_undegraded":"2024-03-28T15:50:37.441703+0000","last_fullsized":"2024-03-28T15:50:37.441703+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:41.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-28T15:50:39.241467+0000","last_change":"2024-03-28T15:50:32.426181+0000","last_active":"2024-03-28T15:50:39.241467+0000","last_peered":"2024-03-28T15:50:39.241467+0000","last_clean":"2024-03-28T15:50:39.241467+0000","last_became_active":"2024-03-28T15:50:32.425866+0000","last_became_peered":"2024-03-28T15:50:32.425866+0000","last_unstale":"2024-03-28T15:50:39.241467+0000","last_undegraded":"2024-03-28T15:50:39.241467+0000","last_fullsized":"2024-03-28T15:50:39.241467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:42.519070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250168,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26819,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":0,"num_read":7,"num_read_kb":9,"num_write":148,"num_write_kb":59,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":43,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":315392,"data_stored":62418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350994,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":485554,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612631,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1097728,"data_stored":1095511,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776156,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":481084,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480892,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1071216,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":480266,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":492460,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":480960,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":484764,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":480550,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888117,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":480910,"data_compressed":14670,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36356,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708796,"statfs":{"total":95995035648,"available":95957807104,"internally_reserved":0,"allocated":1032192,"data_stored":1070816,"data_compressed":51109,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":24877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:51:02.070 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-28T15:51:02.070 INFO:tasks.cephadm:Setting up client nodes... 2024-03-28T15:51:02.071 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-28T15:51:02.356 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:02 smithi023 ceph-mon[32448]: pgmap v303: 129 pgs: 129 active+clean; 603 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-28T15:51:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:02 smithi023 ceph-mon[32448]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:02 smithi023 ceph-mon[32448]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T15:51:02.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:02 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/36091194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:02 smithi117 ceph-mon[36002]: pgmap v303: 129 pgs: 129 active+clean; 603 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-28T15:51:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:02 smithi117 ceph-mon[36002]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:02 smithi117 ceph-mon[36002]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T15:51:02.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:02 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/36091194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:02 smithi120 ceph-mon[36826]: pgmap v303: 129 pgs: 129 active+clean; 603 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-28T15:51:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:02 smithi120 ceph-mon[36826]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:51:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:02 smithi120 ceph-mon[36826]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-28T15:51:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:02 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/36091194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:03.506 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:03 smithi023 ceph-mon[32448]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:03.506 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:03 smithi023 ceph-mon[32448]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:03 smithi117 ceph-mon[36002]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:03.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:03 smithi117 ceph-mon[36002]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:03.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:51:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T15:51:03.256+0000 7f6e41e4c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-28T15:51:03.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:51:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T15:51:03.256+0000 7f6e41e4c700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-28T15:51:03.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 15:51:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T15:51:03.256+0000 7f6e41e4c700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-28T15:51:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:03 smithi120 ceph-mon[36826]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:03.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:03 smithi120 ceph-mon[36826]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:51:04.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 -- 172.21.15.23:0/687282334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c107ec0 msgr2=0x7f409c10a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 --2- 172.21.15.23:0/687282334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c107ec0 0x7f409c10a330 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f409400b0a0 tx=0x7f409400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 -- 172.21.15.23:0/687282334 shutdown_connections 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 --2- 172.21.15.23:0/687282334 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c107ec0 0x7f409c10a330 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 --2- 172.21.15.23:0/687282334 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 0x7f409c105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 --2- 172.21.15.23:0/687282334 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c105590 0x7f409c107980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 -- 172.21.15.23:0/687282334 >> 172.21.15.23:0/687282334 conn(0x7f409c0fc690 msgr2=0x7f409c0feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 -- 172.21.15.23:0/687282334 shutdown_connections 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.267+0000 7f40a3634700 1 -- 172.21.15.23:0/687282334 wait complete. 2024-03-28T15:51:04.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 Processor -- start 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 -- start start 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 0x7f409c198280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 0x7f409c19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10dde0 con 0x7f409c105590 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10dc60 con 0x7f409c107ec0 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a3634700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409c10df60 con 0x7f409c102c60 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a0bcf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a13d0700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 0x7f409c198280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a0bcf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40404/0 (socket says 172.21.15.23:40404) 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a0bcf700 1 -- 172.21.15.23:0/1809362209 learned_addr learned my addr 172.21.15.23:0/1809362209 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.268+0000 7f40a1bd1700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 0x7f409c19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:04.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 -- 172.21.15.23:0/1809362209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 msgr2=0x7f409c198280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 0x7f409c198280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 -- 172.21.15.23:0/1809362209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 msgr2=0x7f409c19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 0x7f409c19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 -- 172.21.15.23:0/1809362209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c19fee0 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a1bd1700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 0x7f409c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a0bcf700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f409000e9d0 tx=0x7f409000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409000cbd0 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f409000ee70 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c100ab0 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4090018660 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.269+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c100ff0 con 0x7f409c105590 2024-03-28T15:51:04.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.270+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c0506a0 con 0x7f409c105590 2024-03-28T15:51:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.271+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4090010430 con 0x7f409c105590 2024-03-28T15:51:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.272+0000 7f408e7fc700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 0x7f40880a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.272+0000 7f40a13d0700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 0x7f40880a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.272+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f40900cf0d0 con 0x7f409c105590 2024-03-28T15:51:04.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.272+0000 7f40a13d0700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 0x7f40880a4e90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f40980044f0 tx=0x7f4098014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:04.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.276+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f409009bdd0 con 0x7f409c105590 2024-03-28T15:51:04.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:04 smithi023 ceph-mon[32448]: pgmap v304: 129 pgs: 129 active+clean; 604 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-28T15:51:04.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.570+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f409c02db00 con 0x7f409c105590 2024-03-28T15:51:04.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:04 smithi117 ceph-mon[36002]: pgmap v304: 129 pgs: 129 active+clean; 604 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-28T15:51:04.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.573+0000 7f408e7fc700 1 -- 172.21.15.23:0/1809362209 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f409c02db00 con 0x7f409c105590 2024-03-28T15:51:04.573 INFO:teuthology.orchestra.run.smithi023.stdout:[client.0] 2024-03-28T15:51:04.573 INFO:teuthology.orchestra.run.smithi023.stdout: key = AQDokQVm0FYTIhAAyerhEtKA6nx2tbxmWWNmww== 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 msgr2=0x7f40880a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 0x7f40880a4e90 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f40980044f0 tx=0x7f4098014040 comp rx=0 tx=0).stop 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 msgr2=0x7f409c1987c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f409000e9d0 tx=0x7f409000ece0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 shutdown_connections 2024-03-28T15:51:04.575 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f40880a29d0 0x7f40880a4e90 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f409c107ec0 0x7f409c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f409c102c60 0x7f409c198280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 --2- 172.21.15.23:0/1809362209 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f409c105590 0x7f409c1987c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 >> 172.21.15.23:0/1809362209 conn(0x7f409c0fc690 msgr2=0x7f409c1035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 shutdown_connections 2024-03-28T15:51:04.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:04.575+0000 7f40a3634700 1 -- 172.21.15.23:0/1809362209 wait complete. 2024-03-28T15:51:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:04 smithi120 ceph-mon[36826]: pgmap v304: 129 pgs: 129 active+clean; 604 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 4.2 KiB/s wr, 11 op/s 2024-03-28T15:51:05.369 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:05 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T15:51:05.369 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:05 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T15:51:05.394 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:51:05.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-28T15:51:05.394 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T15:51:05.436 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T15:51:05.436 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-28T15:51:05.436 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-28T15:51:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:05 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T15:51:05.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:05 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T15:51:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:05 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-28T15:51:05.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:05 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1809362209' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-28T15:51:05.771 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:06.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:06 smithi117 ceph-mon[36002]: pgmap v305: 129 pgs: 129 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.3 KiB/s wr, 19 op/s 2024-03-28T15:51:06.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:06 smithi120 ceph-mon[36826]: pgmap v305: 129 pgs: 129 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.3 KiB/s wr, 19 op/s 2024-03-28T15:51:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:06 smithi023 ceph-mon[32448]: pgmap v305: 129 pgs: 129 active+clean; 606 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6.3 KiB/s wr, 19 op/s 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/2229311436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181021e0 msgr2=0x7f0c18102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/2229311436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181021e0 0x7f0c18102660 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c0099c0 tx=0x7f0c0c009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/2229311436 shutdown_connections 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/2229311436 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/2229311436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181081c0 0x7f0c181085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/2229311436 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181021e0 0x7f0c18102660 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/2229311436 >> 172.21.15.23:0/2229311436 conn(0x7f0c180792c0 msgr2=0x7f0c180796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:07.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/2229311436 shutdown_connections 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.584+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/2229311436 wait complete. 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 Processor -- start 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 -- start start 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 0x7f0c1819e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1819eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c18112440 con 0x7f0c181081c0 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c181122c0 con 0x7f0c181021e0 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c1d6d5700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c181125c0 con 0x7f0c18102ba0 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c177fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40410/0 (socket says 172.21.15.23:40410) 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c177fe700 1 -- 172.21.15.23:0/1916570029 learned_addr learned my addr 172.21.15.23:0/1916570029 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.585+0000 7f0c16ffd700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 0x7f0c1819e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c167fc700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1819eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 -- 172.21.15.23:0/1916570029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 msgr2=0x7f0c1819eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1819eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 -- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 msgr2=0x7f0c18198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 -- 172.21.15.23:0/1916570029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c18199040 con 0x7f0c181021e0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c177fe700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c167fc700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c16ffd700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 0x7f0c1819e640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800e9f0 tx=0x7f0c0800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800c9c0 con 0x7f0c181021e0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c18199330 con 0x7f0c181021e0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c0800cb60 con 0x7f0c181021e0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08018680 con 0x7f0c181021e0 2024-03-28T15:51:07.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.586+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c181a55c0 con 0x7f0c181021e0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.587+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c18103870 con 0x7f0c181021e0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.589+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0c08010430 con 0x7f0c181021e0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.589+0000 7f0bfffff700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 0x7f0c000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.589+0000 7f0c167fc700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 0x7f0c000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.589+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0c080ceee0 con 0x7f0c181021e0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.590+0000 7f0c167fc700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 0x7f0c000a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c18199f60 tx=0x7f0c0c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:07.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.594+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0c0809bbe0 con 0x7f0c181021e0 2024-03-28T15:51:07.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.894+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0c18199c40 con 0x7f0c181021e0 2024-03-28T15:51:07.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.899+0000 7f0bfffff700 1 -- 172.21.15.23:0/1916570029 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262660 (secure 0 0 0) 0x7f0c0809b4e0 con 0x7f0c181021e0 2024-03-28T15:51:07.903 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:07.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 msgr2=0x7f0c000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 0x7f0c000a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0c18199f60 tx=0x7f0c0c005fb0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 msgr2=0x7f0c1819e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:07.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 0x7f0c1819e640 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800e9f0 tx=0x7f0c0800ed00 comp rx=0 tx=0).stop 2024-03-28T15:51:07.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 shutdown_connections 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0c000a2920 0x7f0c000a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0c181081c0 0x7f0c18198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c181021e0 0x7f0c1819e640 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 --2- 172.21.15.23:0/1916570029 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0c18102ba0 0x7f0c1819eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 >> 172.21.15.23:0/1916570029 conn(0x7f0c180792c0 msgr2=0x7f0c18100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 shutdown_connections 2024-03-28T15:51:07.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:07.903+0000 7f0c1d6d5700 1 -- 172.21.15.23:0/1916570029 wait complete. 2024-03-28T15:51:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:08 smithi117 ceph-mon[36002]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T15:51:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:08 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1916570029' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T15:51:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:08 smithi120 ceph-mon[36826]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T15:51:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:08 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1916570029' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T15:51:08.649 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:08 smithi023 ceph-mon[32448]: pgmap v306: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 8.4 KiB/s wr, 22 op/s 2024-03-28T15:51:08.650 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:08 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1916570029' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-28T15:51:08.674 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6800","nonce":1340727098},{"type":"v1","addr":"172.21.15.23:6801","nonce":1340727098}]},"active_addr":"172.21.15.23:6801/1340727098","active_change":"2024-03-28T15:43:34.285518+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.23:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":1859625110}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":3007922889}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":4155829879}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":421542303}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.23:0","nonce":2429196442}]}]} 2024-03-28T15:51:08.679 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-28T15:51:08.679 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-28T15:51:08.679 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T15:51:08.972 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:10.540 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:10 smithi023 ceph-mon[32448]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-28T15:51:10.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:10 smithi117 ceph-mon[36002]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-28T15:51:10.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:10 smithi120 ceph-mon[36826]: pgmap v307: 129 pgs: 129 active+clean; 628 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 8.4 KiB/s wr, 24 op/s 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 -- 172.21.15.23:0/794280312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101a30 msgr2=0x7f3c1810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 --2- 172.21.15.23:0/794280312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101a30 0x7f3c1810df10 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000b0a0 tx=0x7f3c1000b3b0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 -- 172.21.15.23:0/794280312 shutdown_connections 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 --2- 172.21.15.23:0/794280312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101a30 0x7f3c1810df10 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 --2- 172.21.15.23:0/794280312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 0x7f3c181074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 --2- 172.21.15.23:0/794280312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101090 0x7f3c181014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.748+0000 7f3c2072c700 1 -- 172.21.15.23:0/794280312 >> 172.21.15.23:0/794280312 conn(0x7f3c18077fc0 msgr2=0x7f3c180783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- 172.21.15.23:0/794280312 shutdown_connections 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- 172.21.15.23:0/794280312 wait complete. 2024-03-28T15:51:10.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 Processor -- start 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- start start 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 0x7f3c18199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 0x7f3c181a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c18111350 con 0x7f3c18101090 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c181111d0 con 0x7f3c181070d0 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.749+0000 7f3c2072c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c181114d0 con 0x7f3c18101a30 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1e4c8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 0x7f3c18199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1e4c8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48564/0 (socket says 172.21.15.23:48564) 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1e4c8700 1 -- 172.21.15.23:0/3233849027 learned_addr learned my addr 172.21.15.23:0/3233849027 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1ecc9700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 0x7f3c181a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 -- 172.21.15.23:0/3233849027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 msgr2=0x7f3c181a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 0x7f3c181a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 -- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 msgr2=0x7f3c18199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 -- 172.21.15.23:0/3233849027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c181a4d30 con 0x7f3c18101a30 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1e4c8700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c1dcc7700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 0x7f3c18199cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9f0 tx=0x7f3c0800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08004220 con 0x7f3c18101a30 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c181a4f60 con 0x7f3c18101a30 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c08004d10 con 0x7f3c18101a30 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08005020 con 0x7f3c18101a30 2024-03-28T15:51:10.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.750+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c181a5410 con 0x7f3c18101a30 2024-03-28T15:51:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.751+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c18068e50 con 0x7f3c18101a30 2024-03-28T15:51:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.753+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3c08004750 con 0x7f3c18101a30 2024-03-28T15:51:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.754+0000 7f3c0f7fe700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 0x7f3c040a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.754+0000 7f3c1e4c8700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 0x7f3c040a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:10.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.754+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3c080cec90 con 0x7f3c18101a30 2024-03-28T15:51:10.755 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.754+0000 7f3c1e4c8700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 0x7f3c040a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c14011550 tx=0x7f3c14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:10.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.761+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c0809b990 con 0x7f3c18101a30 2024-03-28T15:51:10.966 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.966+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c1802dbf0 con 0x7f3c18101a30 2024-03-28T15:51:10.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.969+0000 7f3c0f7fe700 1 -- 172.21.15.23:0/3233849027 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7f3c0809b290 con 0x7f3c18101a30 2024-03-28T15:51:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:10.969 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:51:10.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 msgr2=0x7f3c040a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 0x7f3c040a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3c14011550 tx=0x7f3c14009260 comp rx=0 tx=0).stop 2024-03-28T15:51:10.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 msgr2=0x7f3c18199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:10.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 0x7f3c18199cb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800e9f0 tx=0x7f3c0800ed00 comp rx=0 tx=0).stop 2024-03-28T15:51:10.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 shutdown_connections 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f3c040a2920 0x7f3c040a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c181070d0 0x7f3c181a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3c18101090 0x7f3c18199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 --2- 172.21.15.23:0/3233849027 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3c18101a30 0x7f3c18199cb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 >> 172.21.15.23:0/3233849027 conn(0x7f3c18077fc0 msgr2=0x7f3c180ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 shutdown_connections 2024-03-28T15:51:10.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:10.971+0000 7f3c2072c700 1 -- 172.21.15.23:0/3233849027 wait complete. 2024-03-28T15:51:11.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:11 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3233849027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:11.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:11 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3233849027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:11.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:11 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3233849027' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:11.845 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-28T15:51:11.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-28T15:51:12.097 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:12.469 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:12 smithi023 ceph-mon[32448]: pgmap v308: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-28T15:51:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:12 smithi117 ceph-mon[36002]: pgmap v308: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-28T15:51:12.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:12 smithi120 ceph-mon[36826]: pgmap v308: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.3 KiB/s wr, 23 op/s 2024-03-28T15:51:13.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 -- 172.21.15.23:0/20107559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101d30 msgr2=0x7f4bdc10df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 --2- 172.21.15.23:0/20107559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101d30 0x7f4bdc10df50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4bd40099c0 tx=0x7f4bd4009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 -- 172.21.15.23:0/20107559 shutdown_connections 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 --2- 172.21.15.23:0/20107559 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101d30 0x7f4bdc10df50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 --2- 172.21.15.23:0/20107559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc1071c0 0x7f4bdc1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 --2- 172.21.15.23:0/20107559 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc101370 0x7f4bdc1017f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 -- 172.21.15.23:0/20107559 >> 172.21.15.23:0/20107559 conn(0x7f4bdc077f50 msgr2=0x7f4bdc078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 -- 172.21.15.23:0/20107559 shutdown_connections 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 -- 172.21.15.23:0/20107559 wait complete. 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.894+0000 7f4be4576700 1 Processor -- start 2024-03-28T15:51:13.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 -- start start 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 0x7f4bdc19eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc1071c0 0x7f4bdc198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc1113c0 con 0x7f4bdc101370 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc111240 con 0x7f4bdc101d30 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be4576700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc111540 con 0x7f4bdc1071c0 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:48590/0 (socket says 172.21.15.23:48590) 2024-03-28T15:51:13.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 -- 172.21.15.23:0/39247703 learned_addr learned my addr 172.21.15.23:0/39247703 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be1b11700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 0x7f4bdc19eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2b13700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc1071c0 0x7f4bdc198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 -- 172.21.15.23:0/39247703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc1071c0 msgr2=0x7f4bdc198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc1071c0 0x7f4bdc198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 -- 172.21.15.23:0/39247703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 msgr2=0x7f4bdc19eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 0x7f4bdc19eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 -- 172.21.15.23:0/39247703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc198f50 con 0x7f4bdc101370 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be1b11700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 0x7f4bdc19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.895+0000 7f4be2312700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00ef10 tx=0x7f4bcc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.896+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00c9c0 con 0x7f4bdc101370 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.896+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4bcc019070 con 0x7f4bdc101370 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.896+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc007e60 con 0x7f4bdc101370 2024-03-28T15:51:13.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.896+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc1991e0 con 0x7f4bdc101370 2024-03-28T15:51:13.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.896+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc1a5520 con 0x7f4bdc101370 2024-03-28T15:51:13.899 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.897+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc1027c0 con 0x7f4bdc101370 2024-03-28T15:51:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.898+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4bcc012050 con 0x7f4bdc101370 2024-03-28T15:51:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.898+0000 7f4bdb7fe700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 0x7f4bc40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.899+0000 7f4be1b11700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 0x7f4bc40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.899+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4bcc0cf420 con 0x7f4bdc101370 2024-03-28T15:51:13.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.899+0000 7f4be1b11700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 0x7f4bc40a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc199e30 tx=0x7f4bd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:13.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:13.904+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4bcc010040 con 0x7f4bdc101370 2024-03-28T15:51:14.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.121+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bdc1a56f0 con 0x7f4bdc101370 2024-03-28T15:51:14.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.123+0000 7f4bdb7fe700 1 -- 172.21.15.23:0/39247703 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+24153 (secure 0 0 0) 0x7f4bcc09c120 con 0x7f4bdc101370 2024-03-28T15:51:14.123 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:51:14.123 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":90,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:50:36.808736+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.125+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 msgr2=0x7f4bc40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.125+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 0x7f4bc40a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc199e30 tx=0x7f4bd0009260 comp rx=0 tx=0).stop 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.125+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 msgr2=0x7f4bdc19e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.125+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00ef10 tx=0x7f4bcc00c3a0 comp rx=0 tx=0).stop 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 shutdown_connections 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4bc40a2920 0x7f4bc40a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4bdc1071c0 0x7f4bdc198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4bdc101370 0x7f4bdc19e5a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:14.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 --2- 172.21.15.23:0/39247703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bdc101d30 0x7f4bdc19eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:14.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 >> 172.21.15.23:0/39247703 conn(0x7f4bdc077f50 msgr2=0x7f4bdc06a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:14.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 shutdown_connections 2024-03-28T15:51:14.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:14.126+0000 7f4be4576700 1 -- 172.21.15.23:0/39247703 wait complete. 2024-03-28T15:51:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:14 smithi023 ceph-mon[32448]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-28T15:51:14.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:14 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/39247703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:14.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:14 smithi117 ceph-mon[36002]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-28T15:51:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:14 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/39247703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:14 smithi120 ceph-mon[36826]: pgmap v309: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.2 KiB/s wr, 18 op/s 2024-03-28T15:51:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:14 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/39247703' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:51:14.938 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-28T15:51:14.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-28T15:51:14.939 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-28T15:51:14.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-28T15:51:14.940 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-28T15:51:14.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-28T15:51:14.941 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-28T15:51:14.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-28T15:51:14.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-28T15:51:14.942 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-28T15:51:14.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-28T15:51:14.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-28T15:51:15.634 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.415 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.416 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.427 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:16 smithi023 ceph-mon[32448]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 18 op/s 2024-03-28T15:51:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.427 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.429 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.429 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.434 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.436 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:16 smithi117 ceph-mon[36002]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 18 op/s 2024-03-28T15:51:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:16 smithi120 ceph-mon[36826]: pgmap v310: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 18 op/s 2024-03-28T15:51:17.167 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:17.549 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:18.496 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:18 smithi023 ceph-mon[32448]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-28T15:51:18.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:18 smithi117 ceph-mon[36002]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-28T15:51:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:18 smithi120 ceph-mon[36826]: pgmap v311: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 3.2 KiB/s wr, 11 op/s 2024-03-28T15:51:20.553 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:20 smithi023 ceph-mon[32448]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T15:51:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:20 smithi117 ceph-mon[36002]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T15:51:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:20 smithi120 ceph-mon[36826]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 -- 172.21.15.23:0/2448806948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc1074a0 msgr2=0x7f7ddc10f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2448806948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc10f050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400b0a0 tx=0x7f7dd400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 -- 172.21.15.23:0/2448806948 shutdown_connections 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2448806948 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc10f050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2448806948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc06a930 0x7f7ddc06ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2448806948 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 0x7f7ddc106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 -- 172.21.15.23:0/2448806948 >> 172.21.15.23:0/2448806948 conn(0x7f7ddc0770e0 msgr2=0x7f7ddc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 -- 172.21.15.23:0/2448806948 shutdown_connections 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.297+0000 7f7de0b55700 1 -- 172.21.15.23:0/2448806948 wait complete. 2024-03-28T15:51:22.336 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 Processor -- start 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 -- start start 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 0x7f7ddc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 0x7f7ddc19e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc1985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc112460 con 0x7f7ddc06a930 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc1122e0 con 0x7f7ddc1074a0 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7de0b55700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc1125e0 con 0x7f7ddc06b2e0 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7dda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 0x7f7ddc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7ddad9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc1985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7dda59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 0x7f7ddc19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60764/0 (socket says 172.21.15.23:60764) 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7dda59c700 1 -- 172.21.15.23:0/2917359697 learned_addr learned my addr 172.21.15.23:0/2917359697 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.298+0000 7f7dd9d9b700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 0x7f7ddc19e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 -- 172.21.15.23:0/2917359697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 msgr2=0x7f7ddc19e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 0x7f7ddc19e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 -- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 msgr2=0x7f7ddc19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 0x7f7ddc19e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc198d40 con 0x7f7ddc1074a0 2024-03-28T15:51:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7dd9d9b700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7ddc06b2e0 0x7f7ddc19e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:22.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7dda59c700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7ddc06a930 0x7f7ddc19e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7ddad9d700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc1985e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400b0a0 tx=0x7f7dd4004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4021070 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dd4007b80 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ddc198fd0 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4005040 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.299+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ddc1a52f0 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.300+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7ddc062920 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.301+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7dd4024070 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dcb7fe700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7dc40a2920 0x7f7dc40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dda59c700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7dc40a2920 0x7f7dc40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7dd400f020 con 0x7f7ddc1074a0 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dcb7fe700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] conn(0x7f7dc40ad5e0 0x7f7dc40afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:22.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dc40b00f0 con 0x7f7dc40ad5e0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd40d0700 con 0x7f7ddc1074a0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dda59c700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7dc40a2920 0x7f7dc40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc011550 tx=0x7f7dcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.302+0000 7f7dd9d9b700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] conn(0x7f7dc40ad5e0 0x7f7dc40afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.303+0000 7f7dd9d9b700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] conn(0x7f7dc40ad5e0 0x7f7dc40afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.303+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== osd.1 v2:172.21.15.117:6800/1603075962 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7dc40b00f0 con 0x7f7dc40ad5e0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 --> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7ddc062da0 con 0x7f7dc40ad5e0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7dcb7fe700 1 -- 172.21.15.23:0/2917359697 <== osd.1 v2:172.21.15.117:6800/1603075962 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7ddc062da0 con 0x7f7dc40ad5e0 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] conn(0x7f7dc40ad5e0 msgr2=0x7f7dc40afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] conn(0x7f7dc40ad5e0 0x7f7dc40afa20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7dc40a2920 msgr2=0x7f7dc40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f7dc40a2920 0x7f7dc40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc011550 tx=0x7f7dcc009260 comp rx=0 tx=0).stop 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc1074a0 msgr2=0x7f7ddc1985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 --2- 172.21.15.23:0/2917359697 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ddc1074a0 0x7f7ddc1985e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400b0a0 tx=0x7f7dd4004d00 comp rx=0 tx=0).stop 2024-03-28T15:51:22.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7ddad9d700 1 -- 172.21.15.23:0/2917359697 reap_dead start 2024-03-28T15:51:22.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 shutdown_connections 2024-03-28T15:51:22.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 >> 172.21.15.23:0/2917359697 conn(0x7f7ddc0770e0 msgr2=0x7f7ddc0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:22.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 shutdown_connections 2024-03-28T15:51:22.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:22.321+0000 7f7de0b55700 1 -- 172.21.15.23:0/2917359697 wait complete. 2024-03-28T15:51:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:22 smithi117 ceph-mon[36002]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T15:51:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:22 smithi120 ceph-mon[36826]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T15:51:22.657 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:22 smithi023 ceph-mon[32448]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-28T15:51:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:24 smithi117 ceph-mon[36002]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T15:51:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:24 smithi120 ceph-mon[36826]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T15:51:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:24 smithi023 ceph-mon[32448]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-28T15:51:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:26 smithi120 ceph-mon[36826]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:26.633 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:26 smithi023 ceph-mon[32448]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:26.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:26 smithi117 ceph-mon[36002]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 -- 172.21.15.23:0/3239483007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09081012f0 msgr2=0x7f09081016d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 --2- 172.21.15.23:0/3239483007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09081012f0 0x7f09081016d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009990 tx=0x7f08f8009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 -- 172.21.15.23:0/3239483007 shutdown_connections 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 --2- 172.21.15.23:0/3239483007 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09081063d0 0x7f090810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 --2- 172.21.15.23:0/3239483007 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09081012f0 0x7f09081016d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 --2- 172.21.15.23:0/3239483007 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0908101ca0 0x7f0908105cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 -- 172.21.15.23:0/3239483007 >> 172.21.15.23:0/3239483007 conn(0x7f0908078160 msgr2=0x7f0908078570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 -- 172.21.15.23:0/3239483007 shutdown_connections 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.460+0000 7f090e4eb700 1 -- 172.21.15.23:0/3239483007 wait complete. 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.461+0000 7f090e4eb700 1 Processor -- start 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.461+0000 7f090e4eb700 1 -- start start 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 0x7f09081997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0908101ca0 0x7f0908199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09081063d0 0x7f09081a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09081113e0 con 0x7f09081012f0 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908111260 con 0x7f0908101ca0 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0908111560 con 0x7f09081063d0 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 0x7f09081997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 0x7f09081997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60788/0 (socket says 172.21.15.23:60788) 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 -- 172.21.15.23:0/1966000343 learned_addr learned my addr 172.21.15.23:0/1966000343 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:27.508 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f09077fe700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0908101ca0 0x7f0908199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090ca88700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09081063d0 0x7f09081a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 -- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09081063d0 msgr2=0x7f09081a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f09081063d0 0x7f09081a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 -- 172.21.15.23:0/1966000343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0908101ca0 msgr2=0x7f0908199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0908101ca0 0x7f0908199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09081a4da0 con 0x7f09081012f0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f0907fff700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 0x7f09081997e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f08f8000c00 tx=0x7f08f800b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8027070 con 0x7f09081012f0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08f8022470 con 0x7f09081012f0 2024-03-28T15:51:27.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.462+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09081a4f70 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.463+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f8017650 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.463+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09081a5310 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.463+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0908062820 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.465+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f08f802a070 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.465+0000 7f09057fa700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f08f00a2920 0x7f08f00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.465+0000 7f09077fe700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f08f00a2920 0x7f08f00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.465+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f08f80d0380 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f09057fa700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] conn(0x7f08f00ad5e0 0x7f08f00afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f08f00b00f0 con 0x7f08f00ad5e0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08f80d07d0 con 0x7f09081012f0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f09077fe700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f08f00a2920 0x7f08f00a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0908105dd0 tx=0x7f08fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f090ca88700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] conn(0x7f08f00ad5e0 0x7f08f00afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:27.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.466+0000 7f090ca88700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] conn(0x7f08f00ad5e0 0x7f08f00afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.467+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== osd.11 v2:172.21.15.120:6824/3052698100 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f08f00b00f0 con 0x7f08f00ad5e0 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.483+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 --> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0908062ca0 con 0x7f08f00ad5e0 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f09057fa700 1 -- 172.21.15.23:0/1966000343 <== osd.11 v2:172.21.15.120:6824/3052698100 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0908062ca0 con 0x7f08f00ad5e0 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] conn(0x7f08f00ad5e0 msgr2=0x7f08f00afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] conn(0x7f08f00ad5e0 0x7f08f00afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f08f00a2920 msgr2=0x7f08f00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f08f00a2920 0x7f08f00a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0908105dd0 tx=0x7f08fc009260 comp rx=0 tx=0).stop 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 msgr2=0x7f09081997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 --2- 172.21.15.23:0/1966000343 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f09081012f0 0x7f09081997e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f08f8000c00 tx=0x7f08f800b610 comp rx=0 tx=0).stop 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090ca88700 1 -- 172.21.15.23:0/1966000343 reap_dead start 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 shutdown_connections 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 >> 172.21.15.23:0/1966000343 conn(0x7f0908078160 msgr2=0x7f0908100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 shutdown_connections 2024-03-28T15:51:27.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:27.484+0000 7f090e4eb700 1 -- 172.21.15.23:0/1966000343 wait complete. 2024-03-28T15:51:28.142 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:28.389 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:28 smithi023 ceph-mon[32448]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:28 smithi120 ceph-mon[36826]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:28 smithi117 ceph-mon[36002]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:29.961 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2519563478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 msgr2=0x7ff7bc10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2519563478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc10f080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400b0d0 tx=0x7ff7b400b3e0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2519563478 shutdown_connections 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2519563478 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc10f080 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2519563478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc1023d0 0x7ff7bc1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2519563478 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7bc102d80 0x7ff7bc106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2519563478 >> 172.21.15.23:0/2519563478 conn(0x7ff7bc0fdc40 msgr2=0x7ff7bc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2519563478 shutdown_connections 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.954+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2519563478 wait complete. 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 Processor -- start 2024-03-28T15:51:29.962 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 -- start start 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7bc1023d0 0x7ff7bc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc102d80 0x7ff7bc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc1987d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc112510 con 0x7ff7bc1074b0 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc112390 con 0x7ff7bc102d80 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7c1b37700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7bc112690 con 0x7ff7bc1023d0 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc1987d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7bbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc1987d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43882/0 (socket says 172.21.15.23:43882) 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7bbfff700 1 -- 172.21.15.23:0/2734541187 learned_addr learned my addr 172.21.15.23:0/2734541187 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7baffd700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc102d80 0x7ff7bc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.955+0000 7ff7bb7fe700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7bc1023d0 0x7ff7bc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 -- 172.21.15.23:0/2734541187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7bc1023d0 msgr2=0x7ff7bc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.963 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff7bc1023d0 0x7ff7bc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 -- 172.21.15.23:0/2734541187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc102d80 msgr2=0x7ff7bc19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc102d80 0x7ff7bc19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc1990b0 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7baffd700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7bc102d80 0x7ff7bc19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7bbfff700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc1987d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400d010 tx=0x7ff7b40079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4003c00 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7b4022070 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7bc199340 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b401b6c0 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.956+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7bc1a5410 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.957+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff7bc0ff140 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.958+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff7b4009ba0 con 0x7ff7bc1074b0 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7b8ff9700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff7a80a29a0 0x7ff7a80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:29.964 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7bb7fe700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff7a80a29a0 0x7ff7a80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff7b40d00f0 con 0x7ff7bc1074b0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7b8ff9700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] conn(0x7ff7a80ad660 0x7ff7a80afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff7a80b0170 con 0x7ff7a80ad660 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff7b40d0540 con 0x7ff7bc1074b0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7baffd700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] conn(0x7ff7a80ad660 0x7ff7a80afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7bb7fe700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff7a80a29a0 0x7ff7a80a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009c60 tx=0x7ff7a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.959+0000 7ff7baffd700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] conn(0x7ff7a80ad660 0x7ff7a80afaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:29.965 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.960+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== osd.9 v2:172.21.15.23:6826/3759918369 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff7a80b0170 con 0x7ff7a80ad660 2024-03-28T15:51:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 --> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff7bc1038b0 con 0x7ff7a80ad660 2024-03-28T15:51:29.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7b8ff9700 1 -- 172.21.15.23:0/2734541187 <== osd.9 v2:172.21.15.23:6826/3759918369 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff7bc1038b0 con 0x7ff7a80ad660 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] conn(0x7ff7a80ad660 msgr2=0x7ff7a80afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] conn(0x7ff7a80ad660 0x7ff7a80afaa0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff7a80a29a0 msgr2=0x7ff7a80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff7a80a29a0 0x7ff7a80a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff7a4009c60 tx=0x7ff7a4009260 comp rx=0 tx=0).stop 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 msgr2=0x7ff7bc1987d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 --2- 172.21.15.23:0/2734541187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff7bc1074b0 0x7ff7bc1987d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400d010 tx=0x7ff7b40079a0 comp rx=0 tx=0).stop 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7bbfff700 1 -- 172.21.15.23:0/2734541187 reap_dead start 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 shutdown_connections 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.977+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 >> 172.21.15.23:0/2734541187 conn(0x7ff7bc0fdc40 msgr2=0x7ff7bc0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:29.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.978+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 shutdown_connections 2024-03-28T15:51:29.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:29.978+0000 7ff7c1b37700 1 -- 172.21.15.23:0/2734541187 wait complete. 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 -- 172.21.15.23:0/113006879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c41023d0 msgr2=0x7fb6c41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 --2- 172.21.15.23:0/113006879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c41023d0 0x7fb6c41027b0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40099c0 tx=0x7fb6b4009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 -- 172.21.15.23:0/113006879 shutdown_connections 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 --2- 172.21.15.23:0/113006879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41074b0 0x7fb6c410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 --2- 172.21.15.23:0/113006879 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c41023d0 0x7fb6c41027b0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 --2- 172.21.15.23:0/113006879 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c4102d80 0x7fb6c4106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 -- 172.21.15.23:0/113006879 >> 172.21.15.23:0/113006879 conn(0x7fb6c40fdc40 msgr2=0x7fb6c4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 -- 172.21.15.23:0/113006879 shutdown_connections 2024-03-28T15:51:30.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.410+0000 7fb6c9083700 1 -- 172.21.15.23:0/113006879 wait complete. 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.411+0000 7fb6c9083700 1 Processor -- start 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.411+0000 7fb6c9083700 1 -- start start 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41023d0 0x7fb6c4074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c4102d80 0x7fb6c40749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 0x7fb6c4074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4112490 con 0x7fb6c4102d80 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4112310 con 0x7fb6c41023d0 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c9083700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c4112610 con 0x7fb6c41074b0 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 0x7fb6c4074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41023d0 0x7fb6c4074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:30.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c359e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 0x7fb6c4074f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:34576/0 (socket says 172.21.15.23:34576) 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c359e700 1 -- 172.21.15.23:0/1371330711 learned_addr learned my addr 172.21.15.23:0/1371330711 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c259c700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c4102d80 0x7fb6c40749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 -- 172.21.15.23:0/1371330711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 msgr2=0x7fb6c4074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 0x7fb6c4074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 -- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c4102d80 msgr2=0x7fb6c40749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c4102d80 0x7fb6c40749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c419ca70 con 0x7fb6c41023d0 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c259c700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb6c4102d80 0x7fb6c40749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c359e700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb6c41074b0 0x7fb6c4074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.412+0000 7fb6c2d9d700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41023d0 0x7fb6c4074490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098e0 tx=0x7fb6b4005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.413+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b4027070 con 0x7fb6c41023d0 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.413+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c419cd00 con 0x7fb6c41023d0 2024-03-28T15:51:30.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.413+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c419d230 con 0x7fb6c41023d0 2024-03-28T15:51:30.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.413+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6b4003680 con 0x7fb6c41023d0 2024-03-28T15:51:30.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.413+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b40173f0 con 0x7fb6c41023d0 2024-03-28T15:51:30.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.414+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6b0001030 con 0x7fb6c41023d0 2024-03-28T15:51:30.415 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.416+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb6b402a070 con 0x7fb6c41023d0 2024-03-28T15:51:30.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.416+0000 7fb6abfff700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6ac0a2a20 0x7fb6ac0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:30.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb6b40cf900 con 0x7fb6c41023d0 2024-03-28T15:51:30.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6c259c700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6ac0a2a20 0x7fb6ac0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:30.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6abfff700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] conn(0x7fb6ac0ad6e0 0x7fb6ac0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:30.417 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6b400dec0 con 0x7fb6ac0ad6e0 2024-03-28T15:51:30.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6b40d42a0 con 0x7fb6c41023d0 2024-03-28T15:51:30.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.417+0000 7fb6c359e700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] conn(0x7fb6ac0ad6e0 0x7fb6ac0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:30.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.418+0000 7fb6c259c700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6ac0a2a20 0x7fb6ac0a4ee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4075c50 tx=0x7fb6b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:30.418 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.418+0000 7fb6c359e700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] conn(0x7fb6ac0ad6e0 0x7fb6ac0afb20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:30.419 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.419+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== osd.10 v2:172.21.15.117:6824/2718715986 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb6b400dec0 con 0x7fb6ac0ad6e0 2024-03-28T15:51:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 --> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb6b0002e60 con 0x7fb6ac0ad6e0 2024-03-28T15:51:30.440 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:30 smithi023 ceph-mon[32448]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:30.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6abfff700 1 -- 172.21.15.23:0/1371330711 <== osd.10 v2:172.21.15.117:6824/2718715986 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb6b0002e60 con 0x7fb6ac0ad6e0 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] conn(0x7fb6ac0ad6e0 msgr2=0x7fb6ac0afb20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6c9083700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] conn(0x7fb6ac0ad6e0 0x7fb6ac0afb20 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6ac0a2a20 msgr2=0x7fb6ac0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.440+0000 7fb6c9083700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb6ac0a2a20 0x7fb6ac0a4ee0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4075c50 tx=0x7fb6b8009260 comp rx=0 tx=0).stop 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41023d0 msgr2=0x7fb6c4074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 --2- 172.21.15.23:0/1371330711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6c41023d0 0x7fb6c4074490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6b40098e0 tx=0x7fb6b4005f10 comp rx=0 tx=0).stop 2024-03-28T15:51:30.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c359e700 1 -- 172.21.15.23:0/1371330711 reap_dead start 2024-03-28T15:51:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 shutdown_connections 2024-03-28T15:51:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 >> 172.21.15.23:0/1371330711 conn(0x7fb6c40fdc40 msgr2=0x7fb6c40ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 shutdown_connections 2024-03-28T15:51:30.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:30.441+0000 7fb6c9083700 1 -- 172.21.15.23:0/1371330711 wait complete. 2024-03-28T15:51:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:30 smithi120 ceph-mon[36826]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:30 smithi117 ceph-mon[36002]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 -- 172.21.15.23:0/3713819272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78881021e0 msgr2=0x7f7888102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 --2- 172.21.15.23:0/3713819272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78881021e0 0x7f7888102660 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f787c0098f0 tx=0x7f787c009c00 comp rx=0 tx=0).stop 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 -- 172.21.15.23:0/3713819272 shutdown_connections 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 --2- 172.21.15.23:0/3713819272 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7888102ba0 0x7f788810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 --2- 172.21.15.23:0/3713819272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78881081c0 0x7f78881085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 --2- 172.21.15.23:0/3713819272 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f78881021e0 0x7f7888102660 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 -- 172.21.15.23:0/3713819272 >> 172.21.15.23:0/3713819272 conn(0x7f78880792c0 msgr2=0x7f78880796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 -- 172.21.15.23:0/3713819272 shutdown_connections 2024-03-28T15:51:31.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.098+0000 7f788cc97700 1 -- 172.21.15.23:0/3713819272 wait complete. 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.099+0000 7f788cc97700 1 Processor -- start 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.099+0000 7f788cc97700 1 -- start start 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78881021e0 0x7f788819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 0x7f788819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881081c0 0x7f7888198850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888112510 con 0x7f78881081c0 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888112390 con 0x7f78881021e0 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788cc97700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7888112690 con 0x7f7888102ba0 2024-03-28T15:51:31.101 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7886d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881081c0 0x7f7888198850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 0x7f788819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f788659c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78881021e0 0x7f788819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 0x7f788819ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:34590/0 (socket says 172.21.15.23:34590) 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 -- 172.21.15.23:0/573888515 learned_addr learned my addr 172.21.15.23:0/573888515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 -- 172.21.15.23:0/573888515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78881021e0 msgr2=0x7f788819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78881021e0 0x7f788819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 -- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881081c0 msgr2=0x7f7888198850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.102 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f78881081c0 0x7f7888198850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888199030 con 0x7f7888102ba0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.100+0000 7f7885d9b700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 0x7f788819ec10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b840 tx=0x7f787c00b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c00bc20 con 0x7f7888102ba0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78881992c0 con 0x7f7888102ba0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f787c00bdc0 con 0x7f7888102ba0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c021420 con 0x7f7888102ba0 2024-03-28T15:51:31.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78881a5530 con 0x7f7888102ba0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.101+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7888062920 con 0x7f7888102ba0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.103+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f787c003710 con 0x7f7888102ba0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.103+0000 7f78777fe700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f78700a29a0 0x7f78700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.103+0000 7f788659c700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f78700a29a0 0x7f78700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.103+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f787c0cfe50 con 0x7f7888102ba0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f78777fe700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] conn(0x7f78700ad660 0x7f78700afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78700b0170 con 0x7f78700ad660 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f787c0d02a0 con 0x7f7888102ba0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f7886d9d700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] conn(0x7f78700ad660 0x7f78700afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f788659c700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f78700a29a0 0x7f78700a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7878009a10 tx=0x7f7878009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.105 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.104+0000 7f7886d9d700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] conn(0x7f78700ad660 0x7f78700afaa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.105+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== osd.8 v2:172.21.15.120:6816/2425649458 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f78700b0170 con 0x7f78700ad660 2024-03-28T15:51:31.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.121+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 --> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7888062d20 con 0x7f78700ad660 2024-03-28T15:51:31.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f78777fe700 1 -- 172.21.15.23:0/573888515 <== osd.8 v2:172.21.15.120:6816/2425649458 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7888062d20 con 0x7f78700ad660 2024-03-28T15:51:31.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] conn(0x7f78700ad660 msgr2=0x7f78700afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] conn(0x7f78700ad660 0x7f78700afaa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f78700a29a0 msgr2=0x7f78700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f78700a29a0 0x7f78700a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7878009a10 tx=0x7f7878009190 comp rx=0 tx=0).stop 2024-03-28T15:51:31.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 msgr2=0x7f788819ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 --2- 172.21.15.23:0/573888515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7888102ba0 0x7f788819ec10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b840 tx=0x7f787c00b870 comp rx=0 tx=0).stop 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f7886d9d700 1 -- 172.21.15.23:0/573888515 reap_dead start 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 shutdown_connections 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 >> 172.21.15.23:0/573888515 conn(0x7f78880792c0 msgr2=0x7f7888100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 shutdown_connections 2024-03-28T15:51:31.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.122+0000 7f788cc97700 1 -- 172.21.15.23:0/573888515 wait complete. 2024-03-28T15:51:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 -- 172.21.15.23:0/3748880070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101090 msgr2=0x7fabb41014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 --2- 172.21.15.23:0/3748880070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101090 0x7fabb41014f0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7faba8009990 tx=0x7faba8009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 -- 172.21.15.23:0/3748880070 shutdown_connections 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 --2- 172.21.15.23:0/3748880070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4101a30 0x7fabb410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 --2- 172.21.15.23:0/3748880070 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb41070d0 0x7fabb41074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 --2- 172.21.15.23:0/3748880070 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101090 0x7fabb41014f0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 -- 172.21.15.23:0/3748880070 >> 172.21.15.23:0/3748880070 conn(0x7fabb4077fc0 msgr2=0x7fabb40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.320+0000 7fabb947b700 1 -- 172.21.15.23:0/3748880070 shutdown_connections 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.321+0000 7fabb947b700 1 -- 172.21.15.23:0/3748880070 wait complete. 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.321+0000 7fabb947b700 1 Processor -- start 2024-03-28T15:51:31.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.321+0000 7fabb947b700 1 -- start start 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb4101090 0x7fabb41997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 0x7fabb4199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb41070d0 0x7fabb41a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb4111380 con 0x7fabb4101a30 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 0x7fabb4199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb27fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 0x7fabb4199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43924/0 (socket says 172.21.15.23:43924) 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb2ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb4101090 0x7fabb41997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb27fc700 1 -- 172.21.15.23:0/3546951669 learned_addr learned my addr 172.21.15.23:0/3546951669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb4111200 con 0x7fabb41070d0 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb4111500 con 0x7fabb4101090 2024-03-28T15:51:31.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb41070d0 0x7fabb41a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 -- 172.21.15.23:0/3546951669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb4101090 msgr2=0x7fabb41997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fabb4101090 0x7fabb41997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 -- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 msgr2=0x7fabb4199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 0x7fabb4199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb37fe700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb41a4d80 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.322+0000 7fabb27fc700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fabb4101a30 0x7fabb4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fabb37fe700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb41070d0 0x7fabb41a4840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabac00efb0 tx=0x7fabac00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac003a00 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabac003ba0 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb41a4fb0 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac004850 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.323+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb41a5470 con 0x7fabb41070d0 2024-03-28T15:51:31.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.324+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fabb40628a0 con 0x7fabb41070d0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.325+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fabac012050 con 0x7fabb41070d0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.325+0000 7fab9bfff700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fab9c0a2920 0x7fab9c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fabb2ffd700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fab9c0a2920 0x7fab9c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fabac0cecb0 con 0x7fabb41070d0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fab9bfff700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] conn(0x7fab9c0ad5e0 0x7fab9c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fab9c0b00f0 con 0x7fab9c0ad5e0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabac0cf060 con 0x7fabb41070d0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fabb27fc700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] conn(0x7fab9c0ad5e0 0x7fab9c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fabb2ffd700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fab9c0a2920 0x7fab9c0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faba4009680 tx=0x7faba4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.326+0000 7fabb27fc700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] conn(0x7fab9c0ad5e0 0x7fab9c0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:31.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.327+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== osd.0 v2:172.21.15.23:6802/3911626101 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fab9c0b00f0 con 0x7fab9c0ad5e0 2024-03-28T15:51:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.344+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 --> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fabb4062d20 con 0x7fab9c0ad5e0 2024-03-28T15:51:31.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.344+0000 7fab9bfff700 1 -- 172.21.15.23:0/3546951669 <== osd.0 v2:172.21.15.23:6802/3911626101 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fabb4062d20 con 0x7fab9c0ad5e0 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] conn(0x7fab9c0ad5e0 msgr2=0x7fab9c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] conn(0x7fab9c0ad5e0 0x7fab9c0afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fab9c0a2920 msgr2=0x7fab9c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fab9c0a2920 0x7fab9c0a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7faba4009680 tx=0x7faba4009260 comp rx=0 tx=0).stop 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb41070d0 msgr2=0x7fabb41a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 --2- 172.21.15.23:0/3546951669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb41070d0 0x7fabb41a4840 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabac00efb0 tx=0x7fabac00c3a0 comp rx=0 tx=0).stop 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb37fe700 1 -- 172.21.15.23:0/3546951669 reap_dead start 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 shutdown_connections 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 >> 172.21.15.23:0/3546951669 conn(0x7fabb4077fc0 msgr2=0x7fabb40ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 shutdown_connections 2024-03-28T15:51:31.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.345+0000 7fabb947b700 1 -- 172.21.15.23:0/3546951669 wait complete. 2024-03-28T15:51:32.082 INFO:teuthology.orchestra.run.smithi023.stdout:347892351000 2024-03-28T15:51:32.082 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-28T15:51:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 -- 172.21.15.23:0/1751515331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101320 msgr2=0x7f57d4101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.197 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 --2- 172.21.15.23:0/1751515331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101320 0x7f57d4101700 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f57d00099c0 tx=0x7f57d0009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 -- 172.21.15.23:0/1751515331 shutdown_connections 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 --2- 172.21.15.23:0/1751515331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4106370 0x7f57d410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 --2- 172.21.15.23:0/1751515331 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101320 0x7f57d4101700 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 --2- 172.21.15.23:0/1751515331 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4101cd0 0x7f57d4105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.749+0000 7f57db439700 1 -- 172.21.15.23:0/1751515331 >> 172.21.15.23:0/1751515331 conn(0x7f57d40770e0 msgr2=0x7f57d40774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.750+0000 7f57db439700 1 -- 172.21.15.23:0/1751515331 shutdown_connections 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.750+0000 7f57db439700 1 -- 172.21.15.23:0/1751515331 wait complete. 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.750+0000 7f57db439700 1 Processor -- start 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.750+0000 7f57db439700 1 -- start start 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4101320 0x7f57d419e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101cd0 0x7f57d419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.198 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4106370 0x7f57d4198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d4111370 con 0x7f57d4101cd0 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d41111f0 con 0x7f57d4101320 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57db439700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d41114f0 con 0x7f57d4106370 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d89d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101cd0 0x7f57d419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4101320 0x7f57d419e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d89d4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101cd0 0x7f57d419ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43942/0 (socket says 172.21.15.23:43942) 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d89d4700 1 -- 172.21.15.23:0/3598571501 learned_addr learned my addr 172.21.15.23:0/3598571501 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d99d6700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4106370 0x7f57d4198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.199 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 -- 172.21.15.23:0/3598571501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4106370 msgr2=0x7f57d4198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57d4106370 0x7f57d4198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 -- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101cd0 msgr2=0x7f57d419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57d4101cd0 0x7f57d419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d4199030 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.751+0000 7f57d91d5700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4101320 0x7f57d419e680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57d00098e0 tx=0x7f57d0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.752+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0027070 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.752+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d41992c0 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.752+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d0003680 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.752+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0022e10 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.752+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d41a5600 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.753+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57d40628a0 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f57d002a070 con 0x7f57d4101320 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c00a29a0 0x7f57c00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.200 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57d89d4700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c00a29a0 0x7f57c00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f57d00cf960 con 0x7f57d4101320 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] conn(0x7f57c00ad660 0x7f57c00afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f57d0001d30 con 0x7f57c00ad660 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.754+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57d00cfd10 con 0x7f57d4101320 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.755+0000 7f57d99d6700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] conn(0x7f57c00ad660 0x7f57c00afaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.755+0000 7f57d89d4700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c00a29a0 0x7f57c00a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f57d4199f10 tx=0x7f57c4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.755+0000 7f57d99d6700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] conn(0x7f57c00ad660 0x7f57c00afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.756+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== osd.6 v2:172.21.15.23:6818/2799436547 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f57d0001d30 con 0x7f57c00ad660 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.772+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 --> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57d4062ca0 con 0x7f57c00ad660 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.772+0000 7f57ca7fc700 1 -- 172.21.15.23:0/3598571501 <== osd.6 v2:172.21.15.23:6818/2799436547 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f57d4062ca0 con 0x7f57c00ad660 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] conn(0x7f57c00ad660 msgr2=0x7f57c00afaa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] conn(0x7f57c00ad660 0x7f57c00afaa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c00a29a0 msgr2=0x7f57c00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c00a29a0 0x7f57c00a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f57d4199f10 tx=0x7f57c4009260 comp rx=0 tx=0).stop 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4101320 msgr2=0x7f57d419e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:32.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 --2- 172.21.15.23:0/3598571501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57d4101320 0x7f57d419e680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57d00098e0 tx=0x7f57d0005f10 comp rx=0 tx=0).stop 2024-03-28T15:51:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57d99d6700 1 -- 172.21.15.23:0/3598571501 reap_dead start 2024-03-28T15:51:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 shutdown_connections 2024-03-28T15:51:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 >> 172.21.15.23:0/3598571501 conn(0x7f57d40770e0 msgr2=0x7f57d40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 shutdown_connections 2024-03-28T15:51:32.202 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:31.773+0000 7f57db439700 1 -- 172.21.15.23:0/3598571501 wait complete. 2024-03-28T15:51:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:32 smithi120 ceph-mon[36826]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:32 smithi023 ceph-mon[32448]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:32 smithi117 ceph-mon[36002]: pgmap v318: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:51:33.099 INFO:teuthology.orchestra.run.smithi023.stdout:292057776163 2024-03-28T15:51:33.100 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-28T15:51:33.535 INFO:teuthology.orchestra.run.smithi023.stdout:257698037800 2024-03-28T15:51:33.535 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-28T15:51:33.635 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:33 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:33.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:33 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:33.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:33 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-28T15:51:34.616 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-28T15:51:34.617 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:34 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-28T15:51:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-28T15:51:34.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:34 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: pgmap v319: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2024-03-28T15:51:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:34 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2024-03-28T15:51:34.934 INFO:teuthology.orchestra.run.smithi023.stdout:201863462962 2024-03-28T15:51:34.934 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 -- 172.21.15.23:0/3001762382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c06aa90 msgr2=0x7fe63c06ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 --2- 172.21.15.23:0/3001762382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c06aa90 0x7fe63c06ae70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe6380099c0 tx=0x7fe638009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 -- 172.21.15.23:0/3001762382 shutdown_connections 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 --2- 172.21.15.23:0/3001762382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c107500 0x7fe63c10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 --2- 172.21.15.23:0/3001762382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c06aa90 0x7fe63c06ae70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 --2- 172.21.15.23:0/3001762382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe63c06b440 0x7fe63c106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 -- 172.21.15.23:0/3001762382 >> 172.21.15.23:0/3001762382 conn(0x7fe63c078000 msgr2=0x7fe63c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 -- 172.21.15.23:0/3001762382 shutdown_connections 2024-03-28T15:51:35.249 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.248+0000 7fe6432b3700 1 -- 172.21.15.23:0/3001762382 wait complete. 2024-03-28T15:51:35.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 Processor -- start 2024-03-28T15:51:35.251 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 -- start start 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe63c06aa90 0x7fe63c101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c06b440 0x7fe63c101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 0x7fe63c101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c112490 con 0x7fe63c107500 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c112310 con 0x7fe63c06b440 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe6432b3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c112610 con 0x7fe63c06aa90 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe641850700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 0x7fe63c101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe641850700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 0x7fe63c101bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43964/0 (socket says 172.21.15.23:43964) 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe641850700 1 -- 172.21.15.23:0/1105531355 learned_addr learned my addr 172.21.15.23:0/1105531355 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.249+0000 7fe64084e700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c06b440 0x7fe63c101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64104f700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe63c06aa90 0x7fe63c101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 -- 172.21.15.23:0/1105531355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe63c06aa90 msgr2=0x7fe63c101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.252 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe63c06aa90 0x7fe63c101150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 -- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 msgr2=0x7fe63c101bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 0x7fe63c101bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c1ab2d0 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe641850700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe63c107500 0x7fe63c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe64084e700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c06b440 0x7fe63c101690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe63c06a6a0 tx=0x7fe62c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c016070 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63c1ab560 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe62c00fd70 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62c019360 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.250+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c1abaf0 con 0x7fe63c06b440 2024-03-28T15:51:35.253 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.251+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe63c0628a0 con 0x7fe63c06b440 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe62c019070 con 0x7fe63c06b440 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6280a2920 0x7fe6280a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe64104f700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6280a2920 0x7fe6280a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe62c0cee90 con 0x7fe63c06b440 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] conn(0x7fe6280ada80 0x7fe6280aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe62c014e80 con 0x7fe6280ada80 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.253+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe62c0cf2e0 con 0x7fe63c06b440 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.254+0000 7fe641850700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] conn(0x7fe6280ada80 0x7fe6280aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.254+0000 7fe64104f700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6280a2920 0x7fe6280a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.254+0000 7fe641850700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] conn(0x7fe6280ada80 0x7fe6280aff20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.255 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.255+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== osd.3 v2:172.21.15.23:6810/4165771629 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe62c014e80 con 0x7fe6280ada80 2024-03-28T15:51:35.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.271+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 --> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe63c062ca0 con 0x7fe6280ada80 2024-03-28T15:51:35.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.271+0000 7fe6327fc700 1 -- 172.21.15.23:0/1105531355 <== osd.3 v2:172.21.15.23:6810/4165771629 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe63c062ca0 con 0x7fe6280ada80 2024-03-28T15:51:35.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] conn(0x7fe6280ada80 msgr2=0x7fe6280aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] conn(0x7fe6280ada80 0x7fe6280aff20 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6280a2920 msgr2=0x7fe6280a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe6280a2920 0x7fe6280a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe638000c00 tx=0x7fe638005fb0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.272 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c06b440 msgr2=0x7fe63c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 --2- 172.21.15.23:0/1105531355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe63c06b440 0x7fe63c101690 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe63c06a6a0 tx=0x7fe62c00dc30 comp rx=0 tx=0).stop 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe641850700 1 -- 172.21.15.23:0/1105531355 reap_dead start 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 shutdown_connections 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 >> 172.21.15.23:0/1105531355 conn(0x7fe63c078000 msgr2=0x7fe63c0ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 shutdown_connections 2024-03-28T15:51:35.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.272+0000 7fe6432b3700 1 -- 172.21.15.23:0/1105531355 wait complete. 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-28T15:51:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:35 smithi023 ceph-mon[32448]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T15:51:35.681 INFO:teuthology.orchestra.run.smithi023.stdout:34359738447 2024-03-28T15:51:35.682 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-28T15:51:35.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:35 smithi117 ceph-mon[36002]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-28T15:51:35.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: osdmap e91: 12 total, 12 up, 12 in 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: pgmap v321: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:51:35.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:35 smithi120 ceph-mon[36826]: osdmap e92: 12 total, 12 up, 12 in 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 -- 172.21.15.23:0/858360999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c1012e0 msgr2=0x7fe85c1016c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 --2- 172.21.15.23:0/858360999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c1012e0 0x7fe85c1016c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe84c009990 tx=0x7fe84c009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 -- 172.21.15.23:0/858360999 shutdown_connections 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 --2- 172.21.15.23:0/858360999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe85c1063a0 0x7fe85c10df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 --2- 172.21.15.23:0/858360999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c1012e0 0x7fe85c1016c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 --2- 172.21.15.23:0/858360999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c101c90 0x7fe85c105cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 -- 172.21.15.23:0/858360999 >> 172.21.15.23:0/858360999 conn(0x7fe85c078000 msgr2=0x7fe85c078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.139+0000 7fe86090e700 1 -- 172.21.15.23:0/858360999 shutdown_connections 2024-03-28T15:51:35.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.140+0000 7fe86090e700 1 -- 172.21.15.23:0/858360999 wait complete. 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.140+0000 7fe86090e700 1 Processor -- start 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.140+0000 7fe86090e700 1 -- start start 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 0x7fe85c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 0x7fe85c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe85c1063a0 0x7fe85c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c111380 con 0x7fe85c101c90 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c111200 con 0x7fe85c1063a0 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe86090e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c111500 con 0x7fe85c1012e0 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe85a59c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 0x7fe85c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 0x7fe85c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe85a59c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 0x7fe85c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:34658/0 (socket says 172.21.15.23:34658) 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe85a59c700 1 -- 172.21.15.23:0/3609573030 learned_addr learned my addr 172.21.15.23:0/3609573030 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:35.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 0x7fe85c19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:43948/0 (socket says 172.21.15.23:43948) 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe85ad9d700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe85c1063a0 0x7fe85c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 -- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 msgr2=0x7fe85c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 0x7fe85c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 -- 172.21.15.23:0/3609573030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe85c1063a0 msgr2=0x7fe85c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe85c1063a0 0x7fe85c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c199040 con 0x7fe85c101c90 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe85a59c700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe85c1012e0 0x7fe85c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.141+0000 7fe859d9b700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 0x7fe85c19ebd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe85000e9c0 tx=0x7fe85000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850004220 con 0x7fe85c101c90 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe86090e700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c199330 con 0x7fe85c101c90 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe850004d10 con 0x7fe85c101c90 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe850005020 con 0x7fe85c101c90 2024-03-28T15:51:35.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe86090e700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c1a5610 con 0x7fe85c101c90 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.142+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe840001030 con 0x7fe85c101c90 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.144+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe850004750 con 0x7fe85c101c90 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.144+0000 7fe84b7fe700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8440a2920 0x7fe8440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.144+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe8500cf270 con 0x7fe85c101c90 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.144+0000 7fe85a59c700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8440a2920 0x7fe8440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe84b7fe700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] conn(0x7fe8440ada80 0x7fe8440aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8440b05f0 con 0x7fe8440ada80 2024-03-28T15:51:35.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8500cf6c0 con 0x7fe85c101c90 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe85ad9d700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] conn(0x7fe8440ada80 0x7fe8440aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe85a59c700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8440a2920 0x7fe8440a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00b2e0 tx=0x7fe84c009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.145+0000 7fe85ad9d700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] conn(0x7fe8440ada80 0x7fe8440aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.147+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== osd.2 v2:172.21.15.120:6800/399817825 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe8440b05f0 con 0x7fe8440ada80 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.163+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 --> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe840002e60 con 0x7fe8440ada80 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe84b7fe700 1 -- 172.21.15.23:0/3609573030 <== osd.2 v2:172.21.15.120:6800/399817825 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe840002e60 con 0x7fe8440ada80 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] conn(0x7fe8440ada80 msgr2=0x7fe8440aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] conn(0x7fe8440ada80 0x7fe8440aff20 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8440a2920 msgr2=0x7fe8440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe8440a2920 0x7fe8440a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe84c00b2e0 tx=0x7fe84c009590 comp rx=0 tx=0).stop 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 msgr2=0x7fe85c19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 --2- 172.21.15.23:0/3609573030 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe85c101c90 0x7fe85c19ebd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe85000e9c0 tx=0x7fe85000ecd0 comp rx=0 tx=0).stop 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe85ad9d700 1 -- 172.21.15.23:0/3609573030 reap_dead start 2024-03-28T15:51:35.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 shutdown_connections 2024-03-28T15:51:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 >> 172.21.15.23:0/3609573030 conn(0x7fe85c078000 msgr2=0x7fe85c06abd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 shutdown_connections 2024-03-28T15:51:35.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:35.164+0000 7fe8497fa700 1 -- 172.21.15.23:0/3609573030 wait complete. 2024-03-28T15:51:37.430 INFO:teuthology.orchestra.run.smithi023.stdout:313532612638 2024-03-28T15:51:37.430 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-28T15:51:37.451 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:37 smithi023 ceph-mon[32448]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T15:51:37.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:37 smithi117 ceph-mon[36002]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T15:51:37.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:37 smithi120 ceph-mon[36826]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s 2024-03-28T15:51:38.497 INFO:teuthology.orchestra.run.smithi023.stdout:55834574921 2024-03-28T15:51:38.497 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-28T15:51:38.914 INFO:teuthology.orchestra.run.smithi023.stdout:111669149762 2024-03-28T15:51:38.915 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-28T15:51:39.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 -- 172.21.15.23:0/320790914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 msgr2=0x7f830810e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 --2- 172.21.15.23:0/320790914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f830810e700 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f8304009990 tx=0x7f8304009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 -- 172.21.15.23:0/320790914 shutdown_connections 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 --2- 172.21.15.23:0/320790914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 0x7f8308111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 --2- 172.21.15.23:0/320790914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f830806a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 --2- 172.21.15.23:0/320790914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f830810e700 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 -- 172.21.15.23:0/320790914 >> 172.21.15.23:0/320790914 conn(0x7f8308077fc0 msgr2=0x7f83080783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 -- 172.21.15.23:0/320790914 shutdown_connections 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.468+0000 7f830fa1c700 1 -- 172.21.15.23:0/320790914 wait complete. 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 Processor -- start 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 -- start start 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f83081a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:39.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f83081a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 0x7f830819aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8308114680 con 0x7f830806a760 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8308114500 con 0x7f8308069e40 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830fa1c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8308114800 con 0x7f830810ec40 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830cfb7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f83081a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830d7b8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f83081a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830d7b8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f83081a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:42214/0 (socket says 172.21.15.23:42214) 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830cfb7700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f83081a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59958/0 (socket says 172.21.15.23:59958) 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830d7b8700 1 -- 172.21.15.23:0/2474293264 learned_addr learned my addr 172.21.15.23:0/2474293264 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.469+0000 7f830dfb9700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 0x7f830819aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 -- 172.21.15.23:0/2474293264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 msgr2=0x7f830819aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 0x7f830819aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 -- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 msgr2=0x7f83081a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f83081a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830819b180 con 0x7f8308069e40 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830dfb9700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f830810ec40 0x7f830819aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830cfb7700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f830806a760 0x7f83081a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830d7b8700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f83081a0810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f82f800e9a0 tx=0x7f82f800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f800ca10 con 0x7f8308069e40 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82f800cbb0 con 0x7f8308069e40 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f8018680 con 0x7f8308069e40 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830819b470 con 0x7f8308069e40 2024-03-28T15:51:39.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.470+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83081a76e0 con 0x7f8308069e40 2024-03-28T15:51:39.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.471+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8308062920 con 0x7f8308069e40 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.472+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f82f8010430 con 0x7f8308069e40 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f82fe7fc700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f82f40a29a0 0x7f82f40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f830cfb7700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f82f40a29a0 0x7f82f40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f82f80cf9c0 con 0x7f8308069e40 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f82fe7fc700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] conn(0x7f82f40adb00 0x7f82f40affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82f801bec0 con 0x7f82f40adb00 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82f80cfd70 con 0x7f8308069e40 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f830cfb7700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f82f40a29a0 0x7f82f40a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f830400b2e0 tx=0x7f8304005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.473+0000 7f830dfb9700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] conn(0x7f82f40adb00 0x7f82f40affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.474+0000 7f830dfb9700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] conn(0x7f82f40adb00 0x7f82f40affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:39.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.475+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== osd.7 v2:172.21.15.117:6816/3265640789 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f82f801bec0 con 0x7f82f40adb00 2024-03-28T15:51:39.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.493+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 --> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8308062d20 con 0x7f82f40adb00 2024-03-28T15:51:39.493 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.493+0000 7f82fe7fc700 1 -- 172.21.15.23:0/2474293264 <== osd.7 v2:172.21.15.117:6816/3265640789 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8308062d20 con 0x7f82f40adb00 2024-03-28T15:51:39.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] conn(0x7f82f40adb00 msgr2=0x7f82f40affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] conn(0x7f82f40adb00 0x7f82f40affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f82f40a29a0 msgr2=0x7f82f40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f82f40a29a0 0x7f82f40a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f830400b2e0 tx=0x7f8304005960 comp rx=0 tx=0).stop 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 msgr2=0x7f83081a0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 --2- 172.21.15.23:0/2474293264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8308069e40 0x7f83081a0810 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f82f800e9a0 tx=0x7f82f800ecb0 comp rx=0 tx=0).stop 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830dfb9700 1 -- 172.21.15.23:0/2474293264 reap_dead start 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 shutdown_connections 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 >> 172.21.15.23:0/2474293264 conn(0x7f8308077fc0 msgr2=0x7f8308110150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 shutdown_connections 2024-03-28T15:51:39.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.494+0000 7f830fa1c700 1 -- 172.21.15.23:0/2474293264 wait complete. 2024-03-28T15:51:39.783 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:39 smithi023 ceph-mon[32448]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:39.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:39 smithi117 ceph-mon[36002]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:39.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:39 smithi120 ceph-mon[36826]: pgmap v324: 129 pgs: 1 peering, 128 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:40.152 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 -- 172.21.15.23:0/2639219734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76741070d0 msgr2=0x7f76741074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 --2- 172.21.15.23:0/2639219734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76741070d0 0x7f76741074b0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7664009990 tx=0x7f7664009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 -- 172.21.15.23:0/2639219734 shutdown_connections 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 --2- 172.21.15.23:0/2639219734 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f7674101a30 0x7f767410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 --2- 172.21.15.23:0/2639219734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76741070d0 0x7f76741074b0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 --2- 172.21.15.23:0/2639219734 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101090 0x7f76741014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 -- 172.21.15.23:0/2639219734 >> 172.21.15.23:0/2639219734 conn(0x7f7674077fc0 msgr2=0x7f76740783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 -- 172.21.15.23:0/2639219734 shutdown_connections 2024-03-28T15:51:40.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.152+0000 7f7679e0b700 1 -- 172.21.15.23:0/2639219734 wait complete. 2024-03-28T15:51:40.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 Processor -- start 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 -- start start 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674101090 0x7f76741a0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101a30 0x7f76741a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 0x7f767419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674111310 con 0x7f76741070d0 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674111190 con 0x7f7674101090 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.153+0000 7f7679e0b700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7674111490 con 0x7f7674101a30 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7673fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 0x7f767419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7673fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 0x7f767419a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59998/0 (socket says 172.21.15.23:59998) 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674101090 0x7f76741a0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7673fff700 1 -- 172.21.15.23:0/1973382766 learned_addr learned my addr 172.21.15.23:0/1973382766 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:40.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7672ffd700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101a30 0x7f76741a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 -- 172.21.15.23:0/1973382766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101a30 msgr2=0x7f76741a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101a30 0x7f76741a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 -- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 msgr2=0x7f767419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 0x7f767419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767419b060 con 0x7f7674101090 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7672ffd700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7674101a30 0x7f76741a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7673fff700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f76741070d0 0x7f767419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f76737fe700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674101090 0x7f76741a0740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76640059d0 tx=0x7f766400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664027070 con 0x7f7674101090 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7664003d70 con 0x7f7674101090 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f767419b2f0 con 0x7f7674101090 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7664022e30 con 0x7f7674101090 2024-03-28T15:51:40.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.154+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76741a7610 con 0x7f7674101090 2024-03-28T15:51:40.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.155+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f767406a8a0 con 0x7f7674101090 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.156+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f766402a070 con 0x7f7674101090 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7670ff9700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f765c0a2920 0x7f765c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7672ffd700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f765c0a2920 0x7f765c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f76640cfea0 con 0x7f7674101090 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7670ff9700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] conn(0x7f765c0ada80 0x7f765c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f765c0b05f0 con 0x7f765c0ada80 2024-03-28T15:51:40.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.157+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76640d02f0 con 0x7f7674101090 2024-03-28T15:51:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.158+0000 7f7673fff700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] conn(0x7f765c0ada80 0x7f765c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.158+0000 7f7672ffd700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f765c0a2920 0x7f765c0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76740783e0 tx=0x7f7668009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.158+0000 7f7673fff700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] conn(0x7f765c0ada80 0x7f765c0aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:40.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.159+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== osd.4 v2:172.21.15.117:6808/1702661707 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f765c0b05f0 con 0x7f765c0ada80 2024-03-28T15:51:40.175 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.175+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 --> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7674102780 con 0x7f765c0ada80 2024-03-28T15:51:40.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7670ff9700 1 -- 172.21.15.23:0/1973382766 <== osd.4 v2:172.21.15.117:6808/1702661707 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7674102780 con 0x7f765c0ada80 2024-03-28T15:51:40.176 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] conn(0x7f765c0ada80 msgr2=0x7f765c0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] conn(0x7f765c0ada80 0x7f765c0aff20 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f765c0a2920 msgr2=0x7f765c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f765c0a2920 0x7f765c0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f76740783e0 tx=0x7f7668009260 comp rx=0 tx=0).stop 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674101090 msgr2=0x7f76741a0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 --2- 172.21.15.23:0/1973382766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7674101090 0x7f76741a0740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f76640059d0 tx=0x7f766400bab0 comp rx=0 tx=0).stop 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7673fff700 1 -- 172.21.15.23:0/1973382766 reap_dead start 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 shutdown_connections 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 >> 172.21.15.23:0/1973382766 conn(0x7f7674077fc0 msgr2=0x7f7674069e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 shutdown_connections 2024-03-28T15:51:40.177 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:40.176+0000 7f7679e0b700 1 -- 172.21.15.23:0/1973382766 wait complete. 2024-03-28T15:51:40.960 INFO:teuthology.orchestra.run.smithi023.stdout:77309411398 2024-03-28T15:51:40.960 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-28T15:51:41.374 INFO:teuthology.orchestra.run.smithi023.stdout:146028888126 2024-03-28T15:51:41.375 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.782+0000 7f958ace3700 1 -- 172.21.15.23:0/1357296371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9584102d80 msgr2=0x7f9584106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.782+0000 7f958ace3700 1 --2- 172.21.15.23:0/1357296371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9584102d80 0x7f9584106dd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f95700099c0 tx=0x7f9570009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 -- 172.21.15.23:0/1357296371 shutdown_connections 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 --2- 172.21.15.23:0/1357296371 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95841074b0 0x7f958410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 --2- 172.21.15.23:0/1357296371 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 --2- 172.21.15.23:0/1357296371 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9584102d80 0x7f9584106dd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 -- 172.21.15.23:0/1357296371 >> 172.21.15.23:0/1357296371 conn(0x7f95840fdc40 msgr2=0x7f9584100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:41.518 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 -- 172.21.15.23:0/1357296371 shutdown_connections 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 -- 172.21.15.23:0/1357296371 wait complete. 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 Processor -- start 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.783+0000 7f958ace3700 1 -- start start 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841a3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9584102d80 0x7f95841a4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95841074b0 0x7f958410d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584112490 con 0x7f95841074b0 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584112310 con 0x7f9584102d80 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f958ace3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9584112610 con 0x7f95841023d0 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841a3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841a3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:34168/0 (socket says 172.21.15.23:34168) 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 -- 172.21.15.23:0/3716498102 learned_addr learned my addr 172.21.15.23:0/3716498102 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9589280700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95841074b0 0x7f958410d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:41.519 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9583fff700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9584102d80 0x7f95841a4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 -- 172.21.15.23:0/3716498102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9584102d80 msgr2=0x7f95841a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9584102d80 0x7f95841a4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 -- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95841074b0 msgr2=0x7f958410d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f95841074b0 0x7f958410d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958410d800 con 0x7f95841023d0 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.784+0000 7f9588a7f700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841a3ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f957800e9c0 tx=0x7f957800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957800cbd0 con 0x7f95841023d0 2024-03-28T15:51:41.520 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f957800ee60 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958410daf0 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9578005630 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95841aba10 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.785+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9584062920 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9578044940 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f95740a29a0 0x7f95740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9583fff700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f95740a29a0 0x7f95740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f95780cf310 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] conn(0x7f95740adb00 0x7f95740affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95740b0670 con 0x7f95740adb00 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95780cf6c0 con 0x7f95841023d0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.787+0000 7f9589280700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] conn(0x7f95740adb00 0x7f95740affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:41.521 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.788+0000 7f9583fff700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f95740a29a0 0x7f95740a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9570000c00 tx=0x7f9570005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.788+0000 7f9589280700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] conn(0x7f95740adb00 0x7f95740affa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.789+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== osd.5 v2:172.21.15.120:6808/3042970364 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f95740b0670 con 0x7f95740adb00 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.805+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 --> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95841038b0 con 0x7f95740adb00 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.805+0000 7f9581ffb700 1 -- 172.21.15.23:0/3716498102 <== osd.5 v2:172.21.15.120:6808/3042970364 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f95841038b0 con 0x7f95740adb00 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.805+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] conn(0x7f95740adb00 msgr2=0x7f95740affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.805+0000 7f958ace3700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] conn(0x7f95740adb00 0x7f95740affa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f95740a29a0 msgr2=0x7f95740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f95740a29a0 0x7f95740a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9570000c00 tx=0x7f9570005fb0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 msgr2=0x7f95841a3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 --2- 172.21.15.23:0/3716498102 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f95841023d0 0x7f95841a3ae0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f957800e9c0 tx=0x7f957800ecd0 comp rx=0 tx=0).stop 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f9589280700 1 -- 172.21.15.23:0/3716498102 reap_dead start 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 shutdown_connections 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 >> 172.21.15.23:0/3716498102 conn(0x7f95840fdc40 msgr2=0x7f95840fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:41.522 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 shutdown_connections 2024-03-28T15:51:41.523 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:39.806+0000 7f958ace3700 1 -- 172.21.15.23:0/3716498102 wait complete. 2024-03-28T15:51:41.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:41 smithi023 ceph-mon[32448]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:41 smithi117 ceph-mon[36002]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:41.838 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:41 smithi120 ceph-mon[36826]: pgmap v325: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:42.188 INFO:teuthology.orchestra.run.smithi023.stdout:167503724601 2024-03-28T15:51:42.188 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-28T15:51:42.671 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.673 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.677 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.680 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.683 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.687 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.694 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:42.752 INFO:teuthology.orchestra.run.smithi023.stdout:223338299439 2024-03-28T15:51:42.752 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-28T15:51:43.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:43 smithi023 ceph-mon[32448]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:43.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:43 smithi117 ceph-mon[36002]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:43.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:43 smithi120 ceph-mon[36826]: pgmap v326: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-28T15:51:44.640 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:45.568 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:45 smithi023 ceph-mon[32448]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T15:51:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:45 smithi117 ceph-mon[36002]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T15:51:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:45 smithi120 ceph-mon[36826]: pgmap v327: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 11 B/s, 0 objects/s recovering 2024-03-28T15:51:46.000 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:46.001 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:47 smithi117 ceph-mon[36002]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 10 B/s, 0 objects/s recovering 2024-03-28T15:51:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:47 smithi120 ceph-mon[36826]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 10 B/s, 0 objects/s recovering 2024-03-28T15:51:47.861 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:47 smithi023 ceph-mon[32448]: pgmap v328: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 10 B/s, 0 objects/s recovering 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 -- 172.21.15.23:0/3599149247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 msgr2=0x7fa568105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/3599149247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa568105d30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa5540099c0 tx=0x7fa554009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 -- 172.21.15.23:0/3599149247 shutdown_connections 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/3599149247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568106410 0x7fa56810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/3599149247 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568101af0 0x7fa568101ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/3599149247 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa568105d30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 -- 172.21.15.23:0/3599149247 >> 172.21.15.23:0/3599149247 conn(0x7fa568077690 msgr2=0x7fa5680feab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 -- 172.21.15.23:0/3599149247 shutdown_connections 2024-03-28T15:51:49.311 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.310+0000 7fa56ecc1700 1 -- 172.21.15.23:0/3599149247 wait complete. 2024-03-28T15:51:49.313 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.311+0000 7fa56ecc1700 1 Processor -- start 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.311+0000 7fa56ecc1700 1 -- start start 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.311+0000 7fa56ecc1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568101af0 0x7fa56819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.311+0000 7fa56ecc1700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.311+0000 7fa56ecc1700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568106410 0x7fa568198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa56ecc1700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5681113d0 con 0x7fa5681024a0 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa56ecc1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa568111250 con 0x7fa568101af0 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa56ecc1700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa568111550 con 0x7fa568106410 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50768/0 (socket says 172.21.15.23:50768) 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 -- 172.21.15.23:0/682386711 learned_addr learned my addr 172.21.15.23:0/682386711 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa56ca5d700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568101af0 0x7fa56819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:49.314 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa56d25e700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568106410 0x7fa568198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 -- 172.21.15.23:0/682386711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568106410 msgr2=0x7fa568198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568106410 0x7fa568198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 -- 172.21.15.23:0/682386711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568101af0 msgr2=0x7fa56819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568101af0 0x7fa56819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 -- 172.21.15.23:0/682386711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa568199000 con 0x7fa5681024a0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa567fff700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa554006010 tx=0x7fa554003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554003680 con 0x7fa5681024a0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5540055a0 con 0x7fa5681024a0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.312+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5540173f0 con 0x7fa5681024a0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.313+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa568199290 con 0x7fa5681024a0 2024-03-28T15:51:49.315 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.313+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5681a54b0 con 0x7fa5681024a0 2024-03-28T15:51:49.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.314+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa568068e50 con 0x7fa5681024a0 2024-03-28T15:51:49.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.315+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa55400f580 con 0x7fa5681024a0 2024-03-28T15:51:49.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.316+0000 7fa565ffb700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 0x7fa5580a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:49.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.316+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa5540cfd00 con 0x7fa5681024a0 2024-03-28T15:51:49.316 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.316+0000 7fa56ca5d700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 0x7fa5580a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:49.317 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.317+0000 7fa56ca5d700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 0x7fa5580a4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009680 tx=0x7fa55c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:49.321 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.321+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa554013070 con 0x7fa5681024a0 2024-03-28T15:51:49.534 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.534+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fa5681a5680 con 0x7fa5681024a0 2024-03-28T15:51:49.535 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.534+0000 7fa565ffb700 1 -- 172.21.15.23:0/682386711 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fa55409c970 con 0x7fa5681024a0 2024-03-28T15:51:49.535 INFO:teuthology.orchestra.run.smithi023.stdout:347892351004 2024-03-28T15:51:49.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 msgr2=0x7fa5580a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:49.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 0x7fa5580a4f80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009680 tx=0x7fa55c009260 comp rx=0 tx=0).stop 2024-03-28T15:51:49.537 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 msgr2=0x7fa56819eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa554006010 tx=0x7fa554003a90 comp rx=0 tx=0).stop 2024-03-28T15:51:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 shutdown_connections 2024-03-28T15:51:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa5580a2ac0 0x7fa5580a4f80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa568106410 0x7fa568198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.538 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa568101af0 0x7fa56819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 --2- 172.21.15.23:0/682386711 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa5681024a0 0x7fa56819eb90 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:49.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 >> 172.21.15.23:0/682386711 conn(0x7fa568077690 msgr2=0x7fa568100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:49.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.537+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 shutdown_connections 2024-03-28T15:51:49.539 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:49.538+0000 7fa56ecc1700 1 -- 172.21.15.23:0/682386711 wait complete. 2024-03-28T15:51:49.552 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:49 smithi023 ceph-mon[32448]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:51:49.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:49 smithi117 ceph-mon[36002]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:51:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:49 smithi120 ceph-mon[36826]: pgmap v329: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:51:50.585 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/682386711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T15:51:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/682386711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T15:51:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/682386711' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 -- 172.21.15.23:0/2194807942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa100101a30 msgr2=0x7fa10010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 --2- 172.21.15.23:0/2194807942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa100101a30 0x7fa10010df10 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800b0a0 tx=0x7fa0f800b3b0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 -- 172.21.15.23:0/2194807942 shutdown_connections 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 --2- 172.21.15.23:0/2194807942 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa100101a30 0x7fa10010df10 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 --2- 172.21.15.23:0/2194807942 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa1001070d0 0x7fa1001074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 --2- 172.21.15.23:0/2194807942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101090 0x7fa1001014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 -- 172.21.15.23:0/2194807942 >> 172.21.15.23:0/2194807942 conn(0x7fa100077fc0 msgr2=0x7fa1000783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:51.036 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 -- 172.21.15.23:0/2194807942 shutdown_connections 2024-03-28T15:51:51.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.034+0000 7fa105239700 1 -- 172.21.15.23:0/2194807942 wait complete. 2024-03-28T15:51:51.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.035+0000 7fa105239700 1 Processor -- start 2024-03-28T15:51:51.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.035+0000 7fa105239700 1 -- start start 2024-03-28T15:51:51.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101a30 0x7fa100103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1001070d0 0x7fa1001037c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa100111370 con 0x7fa1001070d0 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001111f0 con 0x7fa100101a30 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa105239700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1001114f0 con 0x7fa100101090 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:50396/0 (socket says 172.21.15.23:50396) 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 -- 172.21.15.23:0/3516495599 learned_addr learned my addr 172.21.15.23:0/3516495599 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fe59c700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101a30 0x7fa100103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0ff59e700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1001070d0 0x7fa1001037c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 -- 172.21.15.23:0/3516495599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101a30 msgr2=0x7fa100103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101a30 0x7fa100103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 -- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1001070d0 msgr2=0x7fa1001037c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:51.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1001070d0 0x7fa1001037c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 -- 172.21.15.23:0/3516495599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1001a6e10 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.036+0000 7fa0fed9d700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000e9a0 tx=0x7fa0f000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.037+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f000ca10 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.037+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0f000cbb0 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.037+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f0018680 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.037+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1001a7100 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.037+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1001a75f0 con 0x7fa100101090 2024-03-28T15:51:51.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.038+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa100068e50 con 0x7fa100101090 2024-03-28T15:51:51.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.039+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa0f0010430 con 0x7fa100101090 2024-03-28T15:51:51.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.040+0000 7fa0e7fff700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 0x7fa0e80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:51.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.040+0000 7fa0fe59c700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 0x7fa0e80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:51.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.041+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa0f00cf100 con 0x7fa100101090 2024-03-28T15:51:51.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.041+0000 7fa0fe59c700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 0x7fa0e80a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa100078530 tx=0x7fa0f40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.044+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa0f009bc80 con 0x7fa100101090 2024-03-28T15:51:51.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.288+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa1001a7880 con 0x7fa100101090 2024-03-28T15:51:51.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa0e7fff700 1 -- 172.21.15.23:0/3516495599 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa0f009b580 con 0x7fa100101090 2024-03-28T15:51:51.290 INFO:teuthology.orchestra.run.smithi023.stdout:201863462966 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 msgr2=0x7fa0e80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 0x7fa0e80a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa100078530 tx=0x7fa0f40073d0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 msgr2=0x7fa100102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0f000e9a0 tx=0x7fa0f000ecb0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 shutdown_connections 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa0e80a29a0 0x7fa0e80a4e60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa1001070d0 0x7fa1001037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa100101090 0x7fa100102d40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 --2- 172.21.15.23:0/3516495599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa100101a30 0x7fa100103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:51.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 >> 172.21.15.23:0/3516495599 conn(0x7fa100077fc0 msgr2=0x7fa100100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:51.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 shutdown_connections 2024-03-28T15:51:51.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:51.289+0000 7fa105239700 1 -- 172.21.15.23:0/3516495599 wait complete. 2024-03-28T15:51:51.636 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:51 smithi023 ceph-mon[32448]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:51:51.637 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3516495599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T15:51:51.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351000 got 347892351004 for osd.11 2024-03-28T15:51:51.667 DEBUG:teuthology.parallel:result is None 2024-03-28T15:51:51.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:51 smithi117 ceph-mon[36002]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:51:51.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3516495599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T15:51:51.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:51 smithi120 ceph-mon[36826]: pgmap v330: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:51:51.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3516495599' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-28T15:51:53.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:53 smithi023 ceph-mon[32448]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:51:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:53 smithi117 ceph-mon[36002]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:51:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:53 smithi120 ceph-mon[36826]: pgmap v331: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:51:53.937 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462962 got 201863462966 for osd.6 2024-03-28T15:51:53.937 DEBUG:teuthology.parallel:result is None 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- 172.21.15.23:0/2168136983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8108290 msgr2=0x7f92c8108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 --2- 172.21.15.23:0/2168136983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8108290 0x7f92c8108670 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f92c4009990 tx=0x7f92c4009ca0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- 172.21.15.23:0/2168136983 shutdown_connections 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 --2- 172.21.15.23:0/2168136983 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c8102ba0 0x7f92c810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 --2- 172.21.15.23:0/2168136983 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8108290 0x7f92c8108670 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 --2- 172.21.15.23:0/2168136983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c81021e0 0x7f92c8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.120 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- 172.21.15.23:0/2168136983 >> 172.21.15.23:0/2168136983 conn(0x7f92c80792c0 msgr2=0x7f92c80796d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- 172.21.15.23:0/2168136983 shutdown_connections 2024-03-28T15:51:54.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- 172.21.15.23:0/2168136983 wait complete. 2024-03-28T15:51:54.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 Processor -- start 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.119+0000 7f92d0149700 1 -- start start 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c81021e0 0x7f92c819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 0x7f92c8198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c81124d0 con 0x7f92c8102ba0 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8112350 con 0x7f92c8108290 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92d0149700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92c8112650 con 0x7f92c81021e0 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92cd6e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92cd6e4700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50788/0 (socket says 172.21.15.23:50788) 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92cd6e4700 1 -- 172.21.15.23:0/2199469435 learned_addr learned my addr 172.21.15.23:0/2199469435 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 0x7f92c8198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92cdee5700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c81021e0 0x7f92c819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 -- 172.21.15.23:0/2199469435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c81021e0 msgr2=0x7f92c819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c81021e0 0x7f92c819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 -- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 msgr2=0x7f92c819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 -- 172.21.15.23:0/2199469435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92c8198f70 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92cd6e4700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92ce6e6700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 0x7f92c8198790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f92c000efc0 tx=0x7f92c000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.120+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c00092e0 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.121+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92c0004bb0 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.121+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92c8199200 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.121+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92c81a5470 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.121+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0004280 con 0x7f92c8108290 2024-03-28T15:51:54.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.122+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac005330 con 0x7f92c8108290 2024-03-28T15:51:54.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.123+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f92c001b030 con 0x7f92c8108290 2024-03-28T15:51:54.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.123+0000 7f92beffd700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 0x7f92b40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.123+0000 7f92cdee5700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 0x7f92b40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.123+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f92c00ce940 con 0x7f92c8108290 2024-03-28T15:51:54.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.124+0000 7f92cdee5700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 0x7f92b40a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f92c400b2e0 tx=0x7f92c4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.129 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.128+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f92c0022020 con 0x7f92c8108290 2024-03-28T15:51:54.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.354+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f92ac0058f0 con 0x7f92c8108290 2024-03-28T15:51:54.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.354+0000 7f92beffd700 1 -- 172.21.15.23:0/2199469435 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f92c0010040 con 0x7f92c8108290 2024-03-28T15:51:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:257698037804 2024-03-28T15:51:54.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 msgr2=0x7f92b40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 0x7f92b40a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f92c400b2e0 tx=0x7f92c4005960 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 msgr2=0x7f92c8198790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 0x7f92c8198790 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f92c000efc0 tx=0x7f92c000c3a0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 shutdown_connections 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f92b40a29a0 0x7f92b40a4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92c8108290 0x7f92c8198790 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f92c81021e0 0x7f92c819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 --2- 172.21.15.23:0/2199469435 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f92c8102ba0 0x7f92c819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 >> 172.21.15.23:0/2199469435 conn(0x7f92c80792c0 msgr2=0x7f92c8101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 shutdown_connections 2024-03-28T15:51:54.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.357+0000 7f92d0149700 1 -- 172.21.15.23:0/2199469435 wait complete. 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.377+0000 7fb067282700 1 -- 172.21.15.23:0/120618318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060106370 msgr2=0x7fb06010df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 --2- 172.21.15.23:0/120618318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060106370 0x7fb06010df40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b0d0 tx=0x7fb05800b3e0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 -- 172.21.15.23:0/120618318 shutdown_connections 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 --2- 172.21.15.23:0/120618318 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060106370 0x7fb06010df40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 --2- 172.21.15.23:0/120618318 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060101320 0x7fb060101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 --2- 172.21.15.23:0/120618318 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 0x7fb060105c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 -- 172.21.15.23:0/120618318 >> 172.21.15.23:0/120618318 conn(0x7fb0600770e0 msgr2=0x7fb0600774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 -- 172.21.15.23:0/120618318 shutdown_connections 2024-03-28T15:51:54.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 -- 172.21.15.23:0/120618318 wait complete. 2024-03-28T15:51:54.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 Processor -- start 2024-03-28T15:51:54.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.378+0000 7fb067282700 1 -- start start 2024-03-28T15:51:54.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 0x7fb06019e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 0x7fb06019eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 0x7fb0601986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060111360 con 0x7fb060101320 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0601111e0 con 0x7fb060106370 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb067282700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0601114e0 con 0x7fb060101cd0 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06501e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 0x7fb06019e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06501e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 0x7fb06019e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:50802/0 (socket says 172.21.15.23:50802) 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06501e700 1 -- 172.21.15.23:0/522963312 learned_addr learned my addr 172.21.15.23:0/522963312 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 0x7fb0601986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06481d700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 0x7fb06019eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 -- 172.21.15.23:0/522963312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 msgr2=0x7fb06019eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 0x7fb06019eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 -- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 msgr2=0x7fb06019e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 0x7fb06019e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 -- 172.21.15.23:0/522963312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb060198f10 con 0x7fb060106370 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.379+0000 7fb06581f700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 0x7fb0601986b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b770 tx=0x7fb058007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.380+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058021020 con 0x7fb060106370 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.380+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0601991a0 con 0x7fb060106370 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.380+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb058004c50 con 0x7fb060106370 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.380+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb060102e30 con 0x7fb060106370 2024-03-28T15:51:54.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.380+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0580052b0 con 0x7fb060106370 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.381+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb060102770 con 0x7fb060106370 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.382+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb058024020 con 0x7fb060106370 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.382+0000 7fb0567fc700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 0x7fb04c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.383+0000 7fb06501e700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 0x7fb04c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.383+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb05800f020 con 0x7fb060106370 2024-03-28T15:51:54.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.383+0000 7fb06501e700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 0x7fb04c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009680 tx=0x7fb05c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:54.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.391+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb05809d010 con 0x7fb060106370 2024-03-28T15:51:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2199469435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T15:51:54.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.710+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb060199560 con 0x7fb060106370 2024-03-28T15:51:54.711 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.710+0000 7fb0567fc700 1 -- 172.21.15.23:0/522963312 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb05809c910 con 0x7fb060106370 2024-03-28T15:51:54.712 INFO:teuthology.orchestra.run.smithi023.stdout:34359738452 2024-03-28T15:51:54.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 msgr2=0x7fb04c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 0x7fb04c0a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009680 tx=0x7fb05c009260 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 msgr2=0x7fb0601986b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 0x7fb0601986b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b770 tx=0x7fb058007a20 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 shutdown_connections 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb04c0a2920 0x7fb04c0a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb060106370 0x7fb0601986b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb060101320 0x7fb06019e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 --2- 172.21.15.23:0/522963312 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb060101cd0 0x7fb06019eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 >> 172.21.15.23:0/522963312 conn(0x7fb0600770e0 msgr2=0x7fb06006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 shutdown_connections 2024-03-28T15:51:54.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:54.713+0000 7fb067282700 1 -- 172.21.15.23:0/522963312 wait complete. 2024-03-28T15:51:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:54 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2199469435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T15:51:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:54 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2199469435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-28T15:51:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:55 smithi023 ceph-mon[32448]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:51:55.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:55 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/522963312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T15:51:55.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:55 smithi117 ceph-mon[36002]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:51:55.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:55 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/522963312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T15:51:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:55 smithi120 ceph-mon[36826]: pgmap v332: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:51:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:55 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/522963312' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-28T15:51:57.304 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:57.305 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:51:57.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:57 smithi023 ceph-mon[32448]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:51:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:57 smithi117 ceph-mon[36002]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:51:57.838 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:57 smithi120 ceph-mon[36826]: pgmap v333: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:51:57.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037800 got 257698037804 for osd.8 2024-03-28T15:51:57.942 DEBUG:teuthology.parallel:result is None 2024-03-28T15:51:58.381 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738452 for osd.0 2024-03-28T15:51:58.381 DEBUG:teuthology.parallel:result is None 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 -- 172.21.15.23:0/3316505740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806b2e0 msgr2=0x7fd038106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 --2- 172.21.15.23:0/3316505740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806b2e0 0x7fd038106dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd0280098f0 tx=0x7fd028009c00 comp rx=0 tx=0).stop 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 -- 172.21.15.23:0/3316505740 shutdown_connections 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 --2- 172.21.15.23:0/3316505740 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03810f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 --2- 172.21.15.23:0/3316505740 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806a930 0x7fd03806ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 --2- 172.21.15.23:0/3316505740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806b2e0 0x7fd038106dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 -- 172.21.15.23:0/3316505740 >> 172.21.15.23:0/3316505740 conn(0x7fd0380770e0 msgr2=0x7fd0380774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:59.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 -- 172.21.15.23:0/3316505740 shutdown_connections 2024-03-28T15:51:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.053+0000 7fd03f43f700 1 -- 172.21.15.23:0/3316505740 wait complete. 2024-03-28T15:51:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 Processor -- start 2024-03-28T15:51:59.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 -- start start 2024-03-28T15:51:59.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 0x7fd0381a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 0x7fd0381a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03819af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0381016b0 con 0x7fd03806b2e0 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038101530 con 0x7fd03806a930 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03f43f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd038101830 con 0x7fd0381074a0 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03c9da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 0x7fd0381a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03d1db700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 0x7fd0381a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03c9da700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 0x7fd0381a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60730/0 (socket says 172.21.15.23:60730) 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.054+0000 7fd03c9da700 1 -- 172.21.15.23:0/1933373664 learned_addr learned my addr 172.21.15.23:0/1933373664 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d9dc700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03819af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 -- 172.21.15.23:0/1933373664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 msgr2=0x7fd03819af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03819af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 -- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 msgr2=0x7fd0381a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 0x7fd0381a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 -- 172.21.15.23:0/1933373664 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03819b6b0 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d9dc700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03d1db700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 0x7fd0381a0720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9c0 tx=0x7fd03400ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03400c9c0 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd03400cb60 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03819b9a0 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd034018680 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.055+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0381a75f0 con 0x7fd03806a930 2024-03-28T15:51:59.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.056+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd038068e50 con 0x7fd03806a930 2024-03-28T15:51:59.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.058+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd03400cd10 con 0x7fd03806a930 2024-03-28T15:51:59.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.058+0000 7fd02e7fc700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 0x7fd0240a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.058+0000 7fd03c9da700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 0x7fd0240a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.058+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fd0340cf0e0 con 0x7fd03806a930 2024-03-28T15:51:59.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.059+0000 7fd03c9da700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 0x7fd0240a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd0380775e0 tx=0x7fd028019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:59.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.063+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd03409bcc0 con 0x7fd03806a930 2024-03-28T15:51:59.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.288+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fd0381a7820 con 0x7fd03806a930 2024-03-28T15:51:59.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.288+0000 7fd02e7fc700 1 -- 172.21.15.23:0/1933373664 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd03409b5c0 con 0x7fd03806a930 2024-03-28T15:51:59.290 INFO:teuthology.orchestra.run.smithi023.stdout:292057776168 2024-03-28T15:51:59.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 msgr2=0x7fd0240a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 0x7fd0240a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd0380775e0 tx=0x7fd028019040 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 msgr2=0x7fd0381a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 0x7fd0381a0720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9c0 tx=0x7fd03400ecd0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 shutdown_connections 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd0240a2920 0x7fd0240a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd0381074a0 0x7fd03819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd03806a930 0x7fd0381a0720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 --2- 172.21.15.23:0/1933373664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd03806b2e0 0x7fd0381a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 >> 172.21.15.23:0/1933373664 conn(0x7fd0380770e0 msgr2=0x7fd038105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.291+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 shutdown_connections 2024-03-28T15:51:59.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:59.292+0000 7fd03f43f700 1 -- 172.21.15.23:0/1933373664 wait complete. 2024-03-28T15:51:59.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:59 smithi023 ceph-mon[32448]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:51:59.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:59 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/962348973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T15:51:59.582 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:51:59 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1933373664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T15:51:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:59 smithi117 ceph-mon[36002]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:51:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:59 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/962348973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T15:51:59.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:51:59 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1933373664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T15:51:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:59 smithi120 ceph-mon[36826]: pgmap v334: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:51:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:59 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/962348973' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-28T15:51:59.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:51:59 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1933373664' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-28T15:51:59.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 -- 172.21.15.23:0/2941792708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c06b2e0 msgr2=0x7fa75c106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.906 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 --2- 172.21.15.23:0/2941792708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c06b2e0 0x7fa75c106de0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa7500099c0 tx=0x7fa750009cd0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 -- 172.21.15.23:0/2941792708 shutdown_connections 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 --2- 172.21.15.23:0/2941792708 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c107510 0x7fa75c10f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 --2- 172.21.15.23:0/2941792708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c06ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 --2- 172.21.15.23:0/2941792708 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c06b2e0 0x7fa75c106de0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 -- 172.21.15.23:0/2941792708 >> 172.21.15.23:0/2941792708 conn(0x7fa75c0770e0 msgr2=0x7fa75c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 -- 172.21.15.23:0/2941792708 shutdown_connections 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.587+0000 7fa76103f700 1 -- 172.21.15.23:0/2941792708 wait complete. 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 Processor -- start 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 -- start start 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c06b2e0 0x7fa75c19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c112460 con 0x7fa75c107510 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c1122e0 con 0x7fa75c06a930 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa76103f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c1125e0 con 0x7fa75c06b2e0 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.588+0000 7fa75b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75ad9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60710/0 (socket says 172.21.15.23:60710) 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 -- 172.21.15.23:0/962348973 learned_addr learned my addr 172.21.15.23:0/962348973 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75a59c700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c06b2e0 0x7fa75c19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.908 INFO:teuthology.orchestra.run.smithi023.stdout:77309411403 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 -- 172.21.15.23:0/962348973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c06b2e0 msgr2=0x7fa75c19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c06b2e0 0x7fa75c19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 -- 172.21.15.23:0/962348973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 msgr2=0x7fa75c19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 -- 172.21.15.23:0/962348973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c199040 con 0x7fa75c107510 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75ad9d700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa75b59e700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e940 tx=0x7fa75400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754004a30 con 0x7fa75c107510 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c199330 con 0x7fa75c107510 2024-03-28T15:51:59.909 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa754005500 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75400ce20 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.589+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c1a5610 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.590+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c1038c0 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.591+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa754005020 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.592+0000 7fa743fff700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 0x7fa7440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.592+0000 7fa75ad9d700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 0x7fa7440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.592+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa7540ceff0 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.593+0000 7fa75ad9d700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 0x7fa7440a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009480 tx=0x7fa74c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.597+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa75409bbd0 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.809+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa75c199c90 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.809+0000 7fa743fff700 1 -- 172.21.15.23:0/962348973 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa75409b4d0 con 0x7fa75c107510 2024-03-28T15:51:59.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 msgr2=0x7fa7440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 0x7fa7440a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009480 tx=0x7fa74c00b040 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 msgr2=0x7fa75c198760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa75400e940 tx=0x7fa75400ec50 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 shutdown_connections 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa7440a29a0 0x7fa7440a4e60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa75c107510 0x7fa75c198760 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa75c06a930 0x7fa75c19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 --2- 172.21.15.23:0/962348973 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa75c06b2e0 0x7fa75c19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.811+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 >> 172.21.15.23:0/962348973 conn(0x7fa75c0770e0 msgr2=0x7fa75c0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.812+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 shutdown_connections 2024-03-28T15:51:59.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:51:58.812+0000 7fa76103f700 1 -- 172.21.15.23:0/962348973 wait complete. 2024-03-28T15:52:00.555 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-03-28T15:52:00.555 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 -- 172.21.15.23:0/2944037616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 msgr2=0x7f6bcc106dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 --2- 172.21.15.23:0/2944037616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc106dc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0009990 tx=0x7f6bc0009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 -- 172.21.15.23:0/2944037616 shutdown_connections 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 --2- 172.21.15.23:0/2944037616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc1074a0 0x7f6bcc10f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 --2- 172.21.15.23:0/2944037616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc102470 0x7f6bcc102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 --2- 172.21.15.23:0/2944037616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc106dc0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.509 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 -- 172.21.15.23:0/2944037616 >> 172.21.15.23:0/2944037616 conn(0x7f6bcc079220 msgr2=0x7f6bcc079630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 -- 172.21.15.23:0/2944037616 shutdown_connections 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.505+0000 7f6bd2721700 1 -- 172.21.15.23:0/2944037616 wait complete. 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 Processor -- start 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 -- start start 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 0x7f6bcc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc1074a0 0x7f6bcc198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc112440 con 0x7f6bcc102e20 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1122c0 con 0x7f6bcc102470 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bd2721700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bcc1125c0 con 0x7f6bcc1074a0 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:01.510 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bcb7fe700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60766/0 (socket says 172.21.15.23:60766) 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bcb7fe700 1 -- 172.21.15.23:0/432063592 learned_addr learned my addr 172.21.15.23:0/432063592 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bd0cbe700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc1074a0 0x7f6bcc198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.506+0000 7f6bcbfff700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 0x7f6bcc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 -- 172.21.15.23:0/432063592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc1074a0 msgr2=0x7f6bcc198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc1074a0 0x7f6bcc198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 -- 172.21.15.23:0/432063592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 msgr2=0x7f6bcc19e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 0x7f6bcc19e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 -- 172.21.15.23:0/432063592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc199040 con 0x7f6bcc102e20 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcbfff700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 0x7f6bcc19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bcb7fe700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0000c00 tx=0x7f6bc0003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:01.511 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0027070 con 0x7f6bcc102e20 2024-03-28T15:52:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bcc1992d0 con 0x7f6bcc102e20 2024-03-28T15:52:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bc0004050 con 0x7f6bcc102e20 2024-03-28T15:52:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc0022c60 con 0x7f6bcc102e20 2024-03-28T15:52:01.512 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.507+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bcc1a5610 con 0x7f6bcc102e20 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.508+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bcc199630 con 0x7f6bcc102e20 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.510+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6bc002a070 con 0x7f6bcc102e20 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.510+0000 7f6bc97fa700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 0x7f6bb40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.511+0000 7f6bcbfff700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 0x7f6bb40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.511+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f6bc0022e00 con 0x7f6bcc102e20 2024-03-28T15:52:01.513 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.511+0000 7f6bcbfff700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 0x7f6bb40a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc010420 tx=0x7f6bbc0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:01.516 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.515+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6bc0031020 con 0x7f6bcc102e20 2024-03-28T15:52:01.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:01 smithi023 ceph-mon[32448]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:01.610 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:01 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2309841299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T15:52:01.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.776+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6bcc199c90 con 0x7f6bcc102e20 2024-03-28T15:52:01.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.776+0000 7f6bc97fa700 1 -- 172.21.15.23:0/432063592 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6bc0013070 con 0x7f6bcc102e20 2024-03-28T15:52:01.778 INFO:teuthology.orchestra.run.smithi023.stdout:111669149767 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.778+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 msgr2=0x7f6bb40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.778+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 0x7f6bb40a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc010420 tx=0x7f6bbc0073d0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.778+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 msgr2=0x7f6bcc19ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.778+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6bc0000c00 tx=0x7f6bc0003890 comp rx=0 tx=0).stop 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 shutdown_connections 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6bb40a2920 0x7f6bb40a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6bcc1074a0 0x7f6bcc198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bcc102470 0x7f6bcc19e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 --2- 172.21.15.23:0/432063592 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6bcc102e20 0x7f6bcc19ebd0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 >> 172.21.15.23:0/432063592 conn(0x7f6bcc079220 msgr2=0x7f6bcc1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 shutdown_connections 2024-03-28T15:52:01.781 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:01.779+0000 7f6bd2721700 1 -- 172.21.15.23:0/432063592 wait complete. 2024-03-28T15:52:01.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:01 smithi117 ceph-mon[36002]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:01.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:01 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2309841299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T15:52:01.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:01 smithi120 ceph-mon[36826]: pgmap v335: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:01.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:01 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2309841299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-28T15:52:02.615 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:02 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/432063592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T15:52:02.643 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149762 got 111669149767 for osd.3 2024-03-28T15:52:02.643 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:02 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/432063592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T15:52:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:02 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/432063592' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-28T15:52:02.842 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 -- 172.21.15.23:0/3341854314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4601081e0 msgr2=0x7ff4601085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 --2- 172.21.15.23:0/3341854314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4601081e0 0x7ff4601085c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff4500099c0 tx=0x7ff450009cd0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 -- 172.21.15.23:0/3341854314 shutdown_connections 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 --2- 172.21.15.23:0/3341854314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff46006a7e0 0x7ff46010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 --2- 172.21.15.23:0/3341854314 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff4601081e0 0x7ff4601085c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 --2- 172.21.15.23:0/3341854314 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff460069e40 0x7ff46006a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 -- 172.21.15.23:0/3341854314 >> 172.21.15.23:0/3341854314 conn(0x7ff460077fc0 msgr2=0x7ff4600783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 -- 172.21.15.23:0/3341854314 shutdown_connections 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.838+0000 7ff464fb8700 1 -- 172.21.15.23:0/3341854314 wait complete. 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 Processor -- start 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 -- start start 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 0x7ff460101160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.843 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff46006a7e0 0x7ff4601016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 0x7ff460101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff460112440 con 0x7ff460069e40 2024-03-28T15:52:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601122c0 con 0x7ff4601081e0 2024-03-28T15:52:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff464fb8700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4601125c0 con 0x7ff46006a7e0 2024-03-28T15:52:02.844 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.839+0000 7ff45e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 0x7ff460101160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45e59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 0x7ff460101160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60790/0 (socket says 172.21.15.23:60790) 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45e59c700 1 -- 172.21.15.23:0/1089024365 learned_addr learned my addr 172.21.15.23:0/1089024365 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 0x7ff460101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45dd9b700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff46006a7e0 0x7ff4601016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 -- 172.21.15.23:0/1089024365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff46006a7e0 msgr2=0x7ff4601016a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:02.845 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff46006a7e0 0x7ff4601016a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 -- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 msgr2=0x7ff460101160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 0x7ff460101160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 -- 172.21.15.23:0/1089024365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4601a6e70 con 0x7ff4601081e0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff45ed9d700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 0x7ff460101be0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff45800efb0 tx=0x7ff45800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458003a00 con 0x7ff4601081e0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff458003ba0 con 0x7ff4601081e0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff458004850 con 0x7ff4601081e0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.840+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4601a7160 con 0x7ff4601081e0 2024-03-28T15:52:02.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.841+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4601a76a0 con 0x7ff4601081e0 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.842+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff460068e50 con 0x7ff4601081e0 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.843+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff458012050 con 0x7ff4601081e0 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.843+0000 7ff44f7fe700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 0x7ff4480a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.844+0000 7ff45e59c700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 0x7ff4480a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.844+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff4580ce750 con 0x7ff4601081e0 2024-03-28T15:52:02.848 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.844+0000 7ff45e59c700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 0x7ff4480a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff450009990 tx=0x7ff450005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:02.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:02.847+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff458022060 con 0x7ff4601081e0 2024-03-28T15:52:03.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.058+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7ff46002da30 con 0x7ff4601081e0 2024-03-28T15:52:03.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.059+0000 7ff44f7fe700 1 -- 172.21.15.23:0/1089024365 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff458010040 con 0x7ff4601081e0 2024-03-28T15:52:03.060 INFO:teuthology.orchestra.run.smithi023.stdout:167503724606 2024-03-28T15:52:03.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 msgr2=0x7ff4480a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:03.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 0x7ff4480a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff450009990 tx=0x7ff450005960 comp rx=0 tx=0).stop 2024-03-28T15:52:03.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 msgr2=0x7ff460101be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:03.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 0x7ff460101be0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff45800efb0 tx=0x7ff45800c3a0 comp rx=0 tx=0).stop 2024-03-28T15:52:03.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 shutdown_connections 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff4480a29a0 0x7ff4480a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4601081e0 0x7ff460101be0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff460069e40 0x7ff460101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 --2- 172.21.15.23:0/1089024365 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff46006a7e0 0x7ff4601016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 >> 172.21.15.23:0/1089024365 conn(0x7ff460077fc0 msgr2=0x7ff4600ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 shutdown_connections 2024-03-28T15:52:03.064 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:03.062+0000 7ff464fb8700 1 -- 172.21.15.23:0/1089024365 wait complete. 2024-03-28T15:52:03.604 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:03 smithi023 ceph-mon[32448]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:03.604 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:03 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1089024365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T15:52:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:03 smithi117 ceph-mon[36002]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:03.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:03 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1089024365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T15:52:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:03 smithi120 ceph-mon[36826]: pgmap v336: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:03 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1089024365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-28T15:52:04.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724601 got 167503724606 for osd.5 2024-03-28T15:52:04.287 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.740+0000 7f10e3043700 1 -- 172.21.15.23:0/2465099183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06a930 msgr2=0x7f10dc06ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.740+0000 7f10e3043700 1 --2- 172.21.15.23:0/2465099183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06a930 0x7f10dc06ad10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009990 tx=0x7f10d0009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.378 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- 172.21.15.23:0/2465099183 shutdown_connections 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- 172.21.15.23:0/2465099183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc1074a0 0x7f10dc10f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- 172.21.15.23:0/2465099183 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06a930 0x7f10dc06ad10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- 172.21.15.23:0/2465099183 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc06b2e0 0x7f10dc106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- 172.21.15.23:0/2465099183 >> 172.21.15.23:0/2465099183 conn(0x7f10dc0770e0 msgr2=0x7f10dc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- 172.21.15.23:0/2465099183 shutdown_connections 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- 172.21.15.23:0/2465099183 wait complete. 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 Processor -- start 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- start start 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 0x7f10dc101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc1074a0 0x7f10dc101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:04.379 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc112440 con 0x7f10dc06b2e0 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1122c0 con 0x7f10dc06a930 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.741+0000 7f10e3043700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1125c0 con 0x7f10dc1074a0 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 0x7f10dc101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10dbfff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60744/0 (socket says 172.21.15.23:60744) 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10dbfff700 1 -- 172.21.15.23:0/2309841299 learned_addr learned my addr 172.21.15.23:0/2309841299 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e15e0700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc1074a0 0x7f10dc101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 -- 172.21.15.23:0/2309841299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc1074a0 msgr2=0x7f10dc101c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc1074a0 0x7f10dc101c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 -- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 msgr2=0x7f10dc1016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 -- 172.21.15.23:0/2309841299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc1a6d90 con 0x7f10dc06a930 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10dbfff700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:04.380 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e0ddf700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 0x7f10dc101190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d00047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d000f900 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc1a7020 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10d0004830 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.742+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d00054e0 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.743+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc1a7470 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.744+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc068e50 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.745+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f10d0029060 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.745+0000 7f10d9ffb700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 0x7f10cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.745+0000 7f10dbfff700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 0x7f10cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.745+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f10d00cfae0 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.746+0000 7f10dbfff700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 0x7f10cc0a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10dc108970 tx=0x7f10c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.749+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f10d0013070 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.972+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f10dc02dc90 con 0x7f10dc06a930 2024-03-28T15:52:04.381 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.972+0000 7f10d9ffb700 1 -- 172.21.15.23:0/2309841299 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f10d009c750 con 0x7f10dc06a930 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stdout:313532612644 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.974+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 msgr2=0x7f10cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.974+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 0x7f10cc0a4de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f10dc108970 tx=0x7f10c8009260 comp rx=0 tx=0).stop 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.974+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 msgr2=0x7f10dc101190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.974+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 0x7f10dc101190 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f10d0000c00 tx=0x7f10d00047b0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 shutdown_connections 2024-03-28T15:52:04.382 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f10cc0a2920 0x7f10cc0a4de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10dc1074a0 0x7f10dc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10dc06a930 0x7f10dc101190 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 --2- 172.21.15.23:0/2309841299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10dc06b2e0 0x7f10dc1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 >> 172.21.15.23:0/2309841299 conn(0x7f10dc0770e0 msgr2=0x7f10dc0ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 shutdown_connections 2024-03-28T15:52:04.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:00.975+0000 7f10e3043700 1 -- 172.21.15.23:0/2309841299 wait complete. 2024-03-28T15:52:05.419 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612638 got 313532612644 for osd.10 2024-03-28T15:52:05.419 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:05.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 -- 172.21.15.23:0/289409449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b281082b0 msgr2=0x7f6b28108690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:05.607 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/289409449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b281082b0 0x7f6b28108690 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6b18009990 tx=0x7f6b18009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 -- 172.21.15.23:0/289409449 shutdown_connections 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/289409449 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102c00 0x7f6b2810eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/289409449 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b281082b0 0x7f6b28108690 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/289409449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b28102240 0x7f6b281026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 -- 172.21.15.23:0/289409449 >> 172.21.15.23:0/289409449 conn(0x7f6b28079260 msgr2=0x7f6b28079670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 -- 172.21.15.23:0/289409449 shutdown_connections 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.604+0000 7f6b2d95a700 1 -- 172.21.15.23:0/289409449 wait complete. 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 Processor -- start 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 -- start start 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b28102c00 0x7f6b280749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b281082b0 0x7f6b28074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28112500 con 0x7f6b28102c00 2024-03-28T15:52:05.608 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28112380 con 0x7f6b281082b0 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b2d95a700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b28112680 con 0x7f6b28102240 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b267fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b28102c00 0x7f6b280749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b26ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b26ffd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:47712/0 (socket says 172.21.15.23:47712) 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.605+0000 7f6b26ffd700 1 -- 172.21.15.23:0/911177535 learned_addr learned my addr 172.21.15.23:0/911177535 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b277fe700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b281082b0 0x7f6b28074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 -- 172.21.15.23:0/911177535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b281082b0 msgr2=0x7f6b28074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b281082b0 0x7f6b28074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 -- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b28102c00 msgr2=0x7f6b280749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b28102c00 0x7f6b280749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 -- 172.21.15.23:0/911177535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b281a6e90 con 0x7f6b28102240 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b26ffd700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b18000c00 tx=0x7f6b1800bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b18027070 con 0x7f6b28102240 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b18003d70 con 0x7f6b28102240 2024-03-28T15:52:05.609 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b281a7120 con 0x7f6b28102240 2024-03-28T15:52:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1800f460 con 0x7f6b28102240 2024-03-28T15:52:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.606+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b281a7570 con 0x7f6b28102240 2024-03-28T15:52:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.607+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2810a690 con 0x7f6b28102240 2024-03-28T15:52:05.610 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.609+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6b1802a070 con 0x7f6b28102240 2024-03-28T15:52:05.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.609+0000 7f6b2c958700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 0x7f6b100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:05.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.609+0000 7f6b267fc700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 0x7f6b100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:05.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.610+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f6b180cfc20 con 0x7f6b28102240 2024-03-28T15:52:05.611 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.610+0000 7f6b267fc700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 0x7f6b100a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b28079840 tx=0x7f6b1c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:05.614 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.613+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b18013070 con 0x7f6b28102240 2024-03-28T15:52:05.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:05 smithi023 ceph-mon[32448]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:05.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:05 smithi117 ceph-mon[36002]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:05.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.832+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f6b2810a070 con 0x7f6b28102240 2024-03-28T15:52:05.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.832+0000 7f6b2c958700 1 -- 172.21.15.23:0/911177535 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6b1809c990 con 0x7f6b28102240 2024-03-28T15:52:05.834 INFO:teuthology.orchestra.run.smithi023.stdout:55834574929 2024-03-28T15:52:05.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 msgr2=0x7f6b100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:05.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 0x7f6b100a4dd0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6b28079840 tx=0x7f6b1c009260 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 msgr2=0x7f6b28074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6b18000c00 tx=0x7f6b1800bab0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 shutdown_connections 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b100a2910 0x7f6b100a4dd0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b281082b0 0x7f6b28074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b28102240 0x7f6b28074490 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 --2- 172.21.15.23:0/911177535 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b28102c00 0x7f6b280749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 >> 172.21.15.23:0/911177535 conn(0x7f6b28079260 msgr2=0x7f6b28100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 shutdown_connections 2024-03-28T15:52:05.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:05.835+0000 7f6b2d95a700 1 -- 172.21.15.23:0/911177535 wait complete. 2024-03-28T15:52:05.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:05 smithi120 ceph-mon[36826]: pgmap v337: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:06.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:06 smithi023 ceph-mon[32448]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T15:52:06.786 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:06 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/911177535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T15:52:06.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:06 smithi117 ceph-mon[36002]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T15:52:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:06 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/911177535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T15:52:06.838 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:06 smithi120 ceph-mon[36826]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-28T15:52:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:06 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/911177535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-28T15:52:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 -- 172.21.15.23:0/2311977928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641023d0 msgr2=0x7f4b641027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 --2- 172.21.15.23:0/2311977928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641023d0 0x7f4b641027b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4b540099c0 tx=0x7f4b54009cd0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 -- 172.21.15.23:0/2311977928 shutdown_connections 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 --2- 172.21.15.23:0/2311977928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b641074b0 0x7f4b6410f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 --2- 172.21.15.23:0/2311977928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641023d0 0x7f4b641027b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 --2- 172.21.15.23:0/2311977928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b64102d80 0x7f4b64106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 -- 172.21.15.23:0/2311977928 >> 172.21.15.23:0/2311977928 conn(0x7f4b640fdc40 msgr2=0x7f4b64100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 -- 172.21.15.23:0/2311977928 shutdown_connections 2024-03-28T15:52:07.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.106+0000 7f4b68bda700 1 -- 172.21.15.23:0/2311977928 wait complete. 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 Processor -- start 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 -- start start 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 0x7f4b640749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641074b0 0x7f4b64074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64112490 con 0x7f4b641074b0 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64112310 con 0x7f4b641023d0 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.107+0000 7f4b68bda700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b64112610 con 0x7f4b64102d80 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b62d9d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641074b0 0x7f4b64074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b61d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 0x7f4b640749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.110 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:37396/0 (socket says 172.21.15.23:37396) 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 -- 172.21.15.23:0/4042824353 learned_addr learned my addr 172.21.15.23:0/4042824353 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 -- 172.21.15.23:0/4042824353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 msgr2=0x7f4b640749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 0x7f4b640749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 -- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641074b0 msgr2=0x7f4b64074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641074b0 0x7f4b64074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 -- 172.21.15.23:0/4042824353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b641a6e70 con 0x7f4b641023d0 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b61d9b700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 0x7f4b640749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b6259c700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b54006010 tx=0x7f4b54003a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b54003680 con 0x7f4b641023d0 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.108+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b64198680 con 0x7f4b641023d0 2024-03-28T15:52:07.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.109+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b54003dd0 con 0x7f4b641023d0 2024-03-28T15:52:07.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.109+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5400fdf0 con 0x7f4b641023d0 2024-03-28T15:52:07.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.109+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b64198bb0 con 0x7f4b641023d0 2024-03-28T15:52:07.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.110+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b64068e50 con 0x7f4b641023d0 2024-03-28T15:52:07.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.111+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4b5400f4f0 con 0x7f4b641023d0 2024-03-28T15:52:07.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.111+0000 7f4b537fe700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 0x7f4b4c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.111+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f4b540cf9c0 con 0x7f4b641023d0 2024-03-28T15:52:07.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.111+0000 7f4b61d9b700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 0x7f4b4c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.112+0000 7f4b61d9b700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 0x7f4b4c0a4f70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4b64108aa0 tx=0x7f4b58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:07.117 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.116+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4b540d1e20 con 0x7f4b641023d0 2024-03-28T15:52:07.244 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 -- 172.21.15.23:0/2985204047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c10ec70 msgr2=0x7fa29c111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 --2- 172.21.15.23:0/2985204047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c10ec70 0x7fa29c111060 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa29400b0a0 tx=0x7fa29400b3b0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 -- 172.21.15.23:0/2985204047 shutdown_connections 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 --2- 172.21.15.23:0/2985204047 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c10ec70 0x7fa29c111060 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 --2- 172.21.15.23:0/2985204047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c06a6d0 0x7fa29c06aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 --2- 172.21.15.23:0/2985204047 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06aff0 0x7fa29c10e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.994+0000 7fa2a133c700 1 -- 172.21.15.23:0/2985204047 >> 172.21.15.23:0/2985204047 conn(0x7fa29c0770e0 msgr2=0x7fa29c0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.995+0000 7fa2a133c700 1 -- 172.21.15.23:0/2985204047 shutdown_connections 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.995+0000 7fa2a133c700 1 -- 172.21.15.23:0/2985204047 wait complete. 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.995+0000 7fa2a133c700 1 Processor -- start 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.995+0000 7fa2a133c700 1 -- start start 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 0x7fa29c1a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.245 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 0x7fa29c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c114660 con 0x7fa29c06aff0 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c1144e0 con 0x7fa29c10ec70 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa2a133c700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29c1147e0 con 0x7fa29c06a6d0 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29affd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 0x7fa29c1a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60830/0 (socket says 172.21.15.23:60830) 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 -- 172.21.15.23:0/992384105 learned_addr learned my addr 172.21.15.23:0/992384105 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29b7fe700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 0x7fa29c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 -- 172.21.15.23:0/992384105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 msgr2=0x7fa29c1a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 0x7fa29c1a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 -- 172.21.15.23:0/992384105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 msgr2=0x7fa29c19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 0x7fa29c19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.246 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29a7fc700 1 -- 172.21.15.23:0/992384105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c19b100 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29affd700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 0x7fa29c1a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.996+0000 7fa29b7fe700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 0x7fa29c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa29a7fc700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa29000e9c0 tx=0x7fa29000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29000cbd0 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c19b3f0 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa29000ee60 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa290005630 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.997+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c1a7660 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.998+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c068e50 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:06.999+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa290010470 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.000+0000 7fa283fff700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 0x7fa2840a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.000+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fa2900cefe0 con 0x7fa29c06aff0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.000+0000 7fa29affd700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 0x7fa2840a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.000+0000 7fa29affd700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 0x7fa2840a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa29c100de0 tx=0x7fa28c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:07.247 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.004+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa290016330 con 0x7fa29c06aff0 2024-03-28T15:52:07.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.269+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa29c19be80 con 0x7fa29c06aff0 2024-03-28T15:52:07.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.269+0000 7fa283fff700 1 -- 172.21.15.23:0/992384105 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa29009bd20 con 0x7fa29c06aff0 2024-03-28T15:52:07.271 INFO:teuthology.orchestra.run.smithi023.stdout:146028888131 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 msgr2=0x7fa2840a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 0x7fa2840a4d50 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa29c100de0 tx=0x7fa28c009260 comp rx=0 tx=0).stop 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 msgr2=0x7fa29c1a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa29000e9c0 tx=0x7fa29000ecd0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 shutdown_connections 2024-03-28T15:52:07.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa2840a2890 0x7fa2840a4d50 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa29c10ec70 0x7fa29c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa29c06a6d0 0x7fa29c1a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 --2- 172.21.15.23:0/992384105 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa29c06aff0 0x7fa29c1a0cd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 >> 172.21.15.23:0/992384105 conn(0x7fa29c0770e0 msgr2=0x7fa29c1100c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 shutdown_connections 2024-03-28T15:52:07.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.272+0000 7fa2a133c700 1 -- 172.21.15.23:0/992384105 wait complete. 2024-03-28T15:52:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.365+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4b6402da30 con 0x7f4b641023d0 2024-03-28T15:52:07.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.365+0000 7f4b537fe700 1 -- 172.21.15.23:0/4042824353 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4b54013070 con 0x7f4b641023d0 2024-03-28T15:52:07.366 INFO:teuthology.orchestra.run.smithi023.stdout:223338299444 2024-03-28T15:52:07.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 msgr2=0x7f4b4c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 0x7f4b4c0a4f70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f4b64108aa0 tx=0x7f4b58009260 comp rx=0 tx=0).stop 2024-03-28T15:52:07.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 msgr2=0x7f4b64074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:07.369 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4b54006010 tx=0x7f4b54003a90 comp rx=0 tx=0).stop 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 shutdown_connections 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f4b4c0a2ab0 0x7f4b4c0a4f70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4b641074b0 0x7f4b64074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b641023d0 0x7f4b64074490 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 --2- 172.21.15.23:0/4042824353 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4b64102d80 0x7f4b640749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 >> 172.21.15.23:0/4042824353 conn(0x7f4b640fdc40 msgr2=0x7f4b640fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 shutdown_connections 2024-03-28T15:52:07.370 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:07.368+0000 7f4b68bda700 1 -- 172.21.15.23:0/4042824353 wait complete. 2024-03-28T15:52:07.672 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574921 got 55834574929 for osd.1 2024-03-28T15:52:07.672 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:07.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:07 smithi117 ceph-mon[36002]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:52:07.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:07 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/992384105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T15:52:07.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:07 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4042824353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T15:52:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:07 smithi120 ceph-mon[36826]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:52:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:07 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/992384105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T15:52:07.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:07 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4042824353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T15:52:07.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:07 smithi023 ceph-mon[32448]: pgmap v338: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-28T15:52:07.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:07 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/992384105' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-28T15:52:07.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:07 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4042824353' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-28T15:52:08.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776168 for osd.9 2024-03-28T15:52:08.320 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:09.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888126 got 146028888131 for osd.4 2024-03-28T15:52:09.174 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:09.769 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299439 got 223338299444 for osd.7 2024-03-28T15:52:09.769 DEBUG:teuthology.parallel:result is None 2024-03-28T15:52:09.770 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-28T15:52:09.770 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T15:52:09.797 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:09 smithi023 ceph-mon[32448]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:09.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:09 smithi117 ceph-mon[36002]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:09 smithi120 ceph-mon[36826]: pgmap v339: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:10.053 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:10 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:10 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:10 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:11.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- 172.21.15.23:0/923494743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 msgr2=0x7f6540102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:11.774 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 --2- 172.21.15.23:0/923494743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f6540102690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f653c009990 tx=0x7f653c009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- 172.21.15.23:0/923494743 shutdown_connections 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 --2- 172.21.15.23:0/923494743 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6540102bd0 0x7f654010f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 --2- 172.21.15.23:0/923494743 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f65401082f0 0x7f65401086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 --2- 172.21.15.23:0/923494743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f6540102690 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- 172.21.15.23:0/923494743 >> 172.21.15.23:0/923494743 conn(0x7f6540079250 msgr2=0x7f6540079660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- 172.21.15.23:0/923494743 shutdown_connections 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- 172.21.15.23:0/923494743 wait complete. 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 Processor -- start 2024-03-28T15:52:11.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.773+0000 7f6547745700 1 -- start start 2024-03-28T15:52:11.776 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6540102bd0 0x7f65401a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65401082f0 0x7f654019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540112470 con 0x7f65401082f0 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401122f0 con 0x7f6540102210 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6547745700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401125f0 con 0x7f6540102bd0 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6545ce2700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65401082f0 0x7f654019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:38290/0 (socket says 172.21.15.23:38290) 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 -- 172.21.15.23:0/2783788377 learned_addr learned my addr 172.21.15.23:0/2783788377 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f6544ce0700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6540102bd0 0x7f65401a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 -- 172.21.15.23:0/2783788377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6540102bd0 msgr2=0x7f65401a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:11.777 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6540102bd0 0x7f65401a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 -- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65401082f0 msgr2=0x7f654019a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65401082f0 0x7f654019a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 -- 172.21.15.23:0/2783788377 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654019b0b0 con 0x7f6540102210 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65454e1700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9c0 tx=0x7f653000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.774+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000cbd0 con 0x7f6540102210 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.775+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f653000ee50 con 0x7f6540102210 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.775+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654019b3a0 con 0x7f6540102210 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.775+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530005680 con 0x7f6540102210 2024-03-28T15:52:11.778 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.775+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65401a7660 con 0x7f6540102210 2024-03-28T15:52:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.776+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65401038d0 con 0x7f6540102210 2024-03-28T15:52:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.777+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6530004710 con 0x7f6540102210 2024-03-28T15:52:11.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.777+0000 7f65367fc700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 0x7f652c0a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.777+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f65300cefa0 con 0x7f6540102210 2024-03-28T15:52:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.777+0000 7f6544ce0700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 0x7f652c0a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:11.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.778+0000 7f6544ce0700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 0x7f652c0a4c30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2e0 tx=0x7f653c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:11.787 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.786+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f653009bc80 con 0x7f6540102210 2024-03-28T15:52:11.986 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.985+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f654019be80 con 0x7f652c0a2770 2024-03-28T15:52:11.997 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.996+0000 7f65367fc700 1 -- 172.21.15.23:0/2783788377 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340900 (secure 0 0 0) 0x7f654019be80 con 0x7f652c0a2770 2024-03-28T15:52:12.002 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 msgr2=0x7f652c0a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 0x7f652c0a4c30 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f653c00b2e0 tx=0x7f653c005960 comp rx=0 tx=0).stop 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 msgr2=0x7f65401a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9c0 tx=0x7f653000ecd0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 shutdown_connections 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f652c0a2770 0x7f652c0a4c30 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.005 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f65401082f0 0x7f654019a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6540102210 0x7f65401a0790 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 --2- 172.21.15.23:0/2783788377 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6540102bd0 0x7f65401a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 >> 172.21.15.23:0/2783788377 conn(0x7f6540079250 msgr2=0x7f654010d6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 shutdown_connections 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:11.999+0000 7f6547745700 1 -- 172.21.15.23:0/2783788377 wait complete. 2024-03-28T15:52:12.006 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T15:52:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:11 smithi023 ceph-mon[32448]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T15:52:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:11 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:12.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:11 smithi117 ceph-mon[36002]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T15:52:12.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:12.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:12.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:11 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:12.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:11 smithi120 ceph-mon[36826]: pgmap v340: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 11 op/s 2024-03-28T15:52:12.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:52:12.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:52:12.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:11 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' 2024-03-28T15:52:12.815 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2024-03-28T15:52:10.392780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656676,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":870,"num_read_kb":563,"num_write":429,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369864,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571960,"statfs":{"total":1151940427776,"available":1151561687040,"internally_reserved":0,"allocated":12689408,"data_stored":7992446,"data_compressed":357081,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":126,"num_read_kb":71,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003286"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419695+0000","last_change":"2024-03-28T15:50:32.417721+0000","last_active":"2024-03-28T15:52:04.419695+0000","last_peered":"2024-03-28T15:52:04.419695+0000","last_clean":"2024-03-28T15:52:04.419695+0000","last_became_active":"2024-03-28T15:50:32.417397+0000","last_became_peered":"2024-03-28T15:50:32.417397+0000","last_unstale":"2024-03-28T15:52:04.419695+0000","last_undegraded":"2024-03-28T15:52:04.419695+0000","last_fullsized":"2024-03-28T15:52:04.419695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:18.755144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409197+0000","last_change":"2024-03-28T15:50:32.419861+0000","last_active":"2024-03-28T15:52:04.409197+0000","last_peered":"2024-03-28T15:52:04.409197+0000","last_clean":"2024-03-28T15:52:04.409197+0000","last_became_active":"2024-03-28T15:50:32.419472+0000","last_became_peered":"2024-03-28T15:50:32.419472+0000","last_unstale":"2024-03-28T15:52:04.409197+0000","last_undegraded":"2024-03-28T15:52:04.409197+0000","last_fullsized":"2024-03-28T15:52:04.409197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:20.452492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417286+0000","last_change":"2024-03-28T15:50:32.425423+0000","last_active":"2024-03-28T15:52:04.417286+0000","last_peered":"2024-03-28T15:52:04.417286+0000","last_clean":"2024-03-28T15:52:04.417286+0000","last_became_active":"2024-03-28T15:50:32.423503+0000","last_became_peered":"2024-03-28T15:50:32.423503+0000","last_unstale":"2024-03-28T15:52:04.417286+0000","last_undegraded":"2024-03-28T15:52:04.417286+0000","last_fullsized":"2024-03-28T15:52:04.417286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:41.518085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408328+0000","last_change":"2024-03-28T15:50:32.426795+0000","last_active":"2024-03-28T15:52:04.408328+0000","last_peered":"2024-03-28T15:52:04.408328+0000","last_clean":"2024-03-28T15:52:04.408328+0000","last_became_active":"2024-03-28T15:50:32.426495+0000","last_became_peered":"2024-03-28T15:50:32.426495+0000","last_unstale":"2024-03-28T15:52:04.408328+0000","last_undegraded":"2024-03-28T15:52:04.408328+0000","last_fullsized":"2024-03-28T15:52:04.408328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:57:55.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.388993+0000","last_change":"2024-03-28T15:50:32.432258+0000","last_active":"2024-03-28T15:52:04.388993+0000","last_peered":"2024-03-28T15:52:04.388993+0000","last_clean":"2024-03-28T15:52:04.388993+0000","last_became_active":"2024-03-28T15:50:32.431840+0000","last_became_peered":"2024-03-28T15:50:32.431840+0000","last_unstale":"2024-03-28T15:52:04.388993+0000","last_undegraded":"2024-03-28T15:52:04.388993+0000","last_fullsized":"2024-03-28T15:52:04.388993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:30.878251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410749+0000","last_change":"2024-03-28T15:50:32.421362+0000","last_active":"2024-03-28T15:52:04.410749+0000","last_peered":"2024-03-28T15:52:04.410749+0000","last_clean":"2024-03-28T15:52:04.410749+0000","last_became_active":"2024-03-28T15:50:32.421039+0000","last_became_peered":"2024-03-28T15:50:32.421039+0000","last_unstale":"2024-03-28T15:52:04.410749+0000","last_undegraded":"2024-03-28T15:52:04.410749+0000","last_fullsized":"2024-03-28T15:52:04.410749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:01.590154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406923+0000","last_change":"2024-03-28T15:50:32.423923+0000","last_active":"2024-03-28T15:52:04.406923+0000","last_peered":"2024-03-28T15:52:04.406923+0000","last_clean":"2024-03-28T15:52:04.406923+0000","last_became_active":"2024-03-28T15:50:32.423147+0000","last_became_peered":"2024-03-28T15:50:32.423147+0000","last_unstale":"2024-03-28T15:52:04.406923+0000","last_undegraded":"2024-03-28T15:52:04.406923+0000","last_fullsized":"2024-03-28T15:52:04.406923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:42:18.526091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410627+0000","last_change":"2024-03-28T15:50:32.428936+0000","last_active":"2024-03-28T15:52:04.410627+0000","last_peered":"2024-03-28T15:52:04.410627+0000","last_clean":"2024-03-28T15:52:04.410627+0000","last_became_active":"2024-03-28T15:50:32.427438+0000","last_became_peered":"2024-03-28T15:50:32.427438+0000","last_unstale":"2024-03-28T15:52:04.410627+0000","last_undegraded":"2024-03-28T15:52:04.410627+0000","last_fullsized":"2024-03-28T15:52:04.410627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:20.788775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419692+0000","last_change":"2024-03-28T15:50:32.426691+0000","last_active":"2024-03-28T15:52:04.419692+0000","last_peered":"2024-03-28T15:52:04.419692+0000","last_clean":"2024-03-28T15:52:04.419692+0000","last_became_active":"2024-03-28T15:50:32.426408+0000","last_became_peered":"2024-03-28T15:50:32.426408+0000","last_unstale":"2024-03-28T15:52:04.419692+0000","last_undegraded":"2024-03-28T15:52:04.419692+0000","last_fullsized":"2024-03-28T15:52:04.419692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:09:45.911656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409200+0000","last_change":"2024-03-28T15:50:32.428322+0000","last_active":"2024-03-28T15:52:04.409200+0000","last_peered":"2024-03-28T15:52:04.409200+0000","last_clean":"2024-03-28T15:52:04.409200+0000","last_became_active":"2024-03-28T15:50:32.428001+0000","last_became_peered":"2024-03-28T15:50:32.428001+0000","last_unstale":"2024-03-28T15:52:04.409200+0000","last_undegraded":"2024-03-28T15:52:04.409200+0000","last_fullsized":"2024-03-28T15:52:04.409200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:00.795303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417305+0000","last_change":"2024-03-28T15:50:32.435072+0000","last_active":"2024-03-28T15:52:04.417305+0000","last_peered":"2024-03-28T15:52:04.417305+0000","last_clean":"2024-03-28T15:52:04.417305+0000","last_became_active":"2024-03-28T15:50:32.434757+0000","last_became_peered":"2024-03-28T15:50:32.434757+0000","last_unstale":"2024-03-28T15:52:04.417305+0000","last_undegraded":"2024-03-28T15:52:04.417305+0000","last_fullsized":"2024-03-28T15:52:04.417305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:34.357151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.405151+0000","last_change":"2024-03-28T15:50:32.428312+0000","last_active":"2024-03-28T15:52:04.405151+0000","last_peered":"2024-03-28T15:52:04.405151+0000","last_clean":"2024-03-28T15:52:04.405151+0000","last_became_active":"2024-03-28T15:50:32.428020+0000","last_became_peered":"2024-03-28T15:50:32.428020+0000","last_unstale":"2024-03-28T15:52:04.405151+0000","last_undegraded":"2024-03-28T15:52:04.405151+0000","last_fullsized":"2024-03-28T15:52:04.405151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:11.841606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398999+0000","last_change":"2024-03-28T15:50:32.422259+0000","last_active":"2024-03-28T15:52:04.398999+0000","last_peered":"2024-03-28T15:52:04.398999+0000","last_clean":"2024-03-28T15:52:04.398999+0000","last_became_active":"2024-03-28T15:50:32.421181+0000","last_became_peered":"2024-03-28T15:50:32.421181+0000","last_unstale":"2024-03-28T15:52:04.398999+0000","last_undegraded":"2024-03-28T15:52:04.398999+0000","last_fullsized":"2024-03-28T15:52:04.398999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:40.403642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406213+0000","last_change":"2024-03-28T15:50:32.430402+0000","last_active":"2024-03-28T15:52:04.406213+0000","last_peered":"2024-03-28T15:52:04.406213+0000","last_clean":"2024-03-28T15:52:04.406213+0000","last_became_active":"2024-03-28T15:50:32.429978+0000","last_became_peered":"2024-03-28T15:50:32.429978+0000","last_unstale":"2024-03-28T15:52:04.406213+0000","last_undegraded":"2024-03-28T15:52:04.406213+0000","last_fullsized":"2024-03-28T15:52:04.406213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:06.407373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410258+0000","last_change":"2024-03-28T15:51:35.358518+0000","last_active":"2024-03-28T15:52:04.410258+0000","last_peered":"2024-03-28T15:52:04.410258+0000","last_clean":"2024-03-28T15:52:04.410258+0000","last_became_active":"2024-03-28T15:51:35.358203+0000","last_became_peered":"2024-03-28T15:51:35.358203+0000","last_unstale":"2024-03-28T15:52:04.410258+0000","last_undegraded":"2024-03-28T15:52:04.410258+0000","last_fullsized":"2024-03-28T15:52:04.410258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:04.984476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419377+0000","last_change":"2024-03-28T15:50:32.429542+0000","last_active":"2024-03-28T15:52:04.419377+0000","last_peered":"2024-03-28T15:52:04.419377+0000","last_clean":"2024-03-28T15:52:04.419377+0000","last_became_active":"2024-03-28T15:50:32.427264+0000","last_became_peered":"2024-03-28T15:50:32.427264+0000","last_unstale":"2024-03-28T15:52:04.419377+0000","last_undegraded":"2024-03-28T15:52:04.419377+0000","last_fullsized":"2024-03-28T15:52:04.419377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.272678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.420043+0000","last_change":"2024-03-28T15:50:32.418775+0000","last_active":"2024-03-28T15:52:04.420043+0000","last_peered":"2024-03-28T15:52:04.420043+0000","last_clean":"2024-03-28T15:52:04.420043+0000","last_became_active":"2024-03-28T15:50:32.418453+0000","last_became_peered":"2024-03-28T15:50:32.418453+0000","last_unstale":"2024-03-28T15:52:04.420043+0000","last_undegraded":"2024-03-28T15:52:04.420043+0000","last_fullsized":"2024-03-28T15:52:04.420043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:26.061231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402093+0000","last_change":"2024-03-28T15:50:32.429111+0000","last_active":"2024-03-28T15:52:04.402093+0000","last_peered":"2024-03-28T15:52:04.402093+0000","last_clean":"2024-03-28T15:52:04.402093+0000","last_became_active":"2024-03-28T15:50:32.428036+0000","last_became_peered":"2024-03-28T15:50:32.428036+0000","last_unstale":"2024-03-28T15:52:04.402093+0000","last_undegraded":"2024-03-28T15:52:04.402093+0000","last_fullsized":"2024-03-28T15:52:04.402093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:09.272224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.393050+0000","last_change":"2024-03-28T15:50:32.422982+0000","last_active":"2024-03-28T15:52:04.393050+0000","last_peered":"2024-03-28T15:52:04.393050+0000","last_clean":"2024-03-28T15:52:04.393050+0000","last_became_active":"2024-03-28T15:50:32.422667+0000","last_became_peered":"2024-03-28T15:50:32.422667+0000","last_unstale":"2024-03-28T15:52:04.393050+0000","last_undegraded":"2024-03-28T15:52:04.393050+0000","last_fullsized":"2024-03-28T15:52:04.393050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:51.445603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351583+0000","last_change":"2024-03-28T15:50:36.462626+0000","last_active":"2024-03-28T15:51:35.351583+0000","last_peered":"2024-03-28T15:51:35.351583+0000","last_clean":"2024-03-28T15:51:35.351583+0000","last_became_active":"2024-03-28T15:50:36.462168+0000","last_became_peered":"2024-03-28T15:50:36.462168+0000","last_unstale":"2024-03-28T15:51:35.351583+0000","last_undegraded":"2024-03-28T15:51:35.351583+0000","last_fullsized":"2024-03-28T15:51:35.351583+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:59.676350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:07.301680+0000","last_change":"2024-03-28T15:50:31.425778+0000","last_active":"2024-03-28T15:52:07.301680+0000","last_peered":"2024-03-28T15:52:07.301680+0000","last_clean":"2024-03-28T15:52:07.301680+0000","last_became_active":"2024-03-28T15:50:31.424946+0000","last_became_peered":"2024-03-28T15:50:31.424946+0000","last_unstale":"2024-03-28T15:52:07.301680+0000","last_undegraded":"2024-03-28T15:52:07.301680+0000","last_fullsized":"2024-03-28T15:52:07.301680+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:50.444459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.401877+0000","last_change":"2024-03-28T15:50:32.431201+0000","last_active":"2024-03-28T15:52:04.401877+0000","last_peered":"2024-03-28T15:52:04.401877+0000","last_clean":"2024-03-28T15:52:04.401877+0000","last_became_active":"2024-03-28T15:50:32.428970+0000","last_became_peered":"2024-03-28T15:50:32.428970+0000","last_unstale":"2024-03-28T15:52:04.401877+0000","last_undegraded":"2024-03-28T15:52:04.401877+0000","last_fullsized":"2024-03-28T15:52:04.401877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:38:36.966552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349277+0000","last_change":"2024-03-28T15:50:36.457727+0000","last_active":"2024-03-28T15:51:35.349277+0000","last_peered":"2024-03-28T15:51:35.349277+0000","last_clean":"2024-03-28T15:51:35.349277+0000","last_became_active":"2024-03-28T15:50:36.457322+0000","last_became_peered":"2024-03-28T15:50:36.457322+0000","last_unstale":"2024-03-28T15:51:35.349277+0000","last_undegraded":"2024-03-28T15:51:35.349277+0000","last_fullsized":"2024-03-28T15:51:35.349277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:09.138564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.852963+0000","last_change":"2024-03-28T15:50:31.418890+0000","last_active":"2024-03-28T15:51:35.852963+0000","last_peered":"2024-03-28T15:51:35.852963+0000","last_clean":"2024-03-28T15:51:35.852963+0000","last_became_active":"2024-03-28T15:50:31.418445+0000","last_became_peered":"2024-03-28T15:50:31.418445+0000","last_unstale":"2024-03-28T15:51:35.852963+0000","last_undegraded":"2024-03-28T15:51:35.852963+0000","last_fullsized":"2024-03-28T15:51:35.852963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:15.157292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419869+0000","last_change":"2024-03-28T15:50:32.434362+0000","last_active":"2024-03-28T15:52:04.419869+0000","last_peered":"2024-03-28T15:52:04.419869+0000","last_clean":"2024-03-28T15:52:04.419869+0000","last_became_active":"2024-03-28T15:50:32.433577+0000","last_became_peered":"2024-03-28T15:50:32.433577+0000","last_unstale":"2024-03-28T15:52:04.419869+0000","last_undegraded":"2024-03-28T15:52:04.419869+0000","last_fullsized":"2024-03-28T15:52:04.419869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:13.424149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349380+0000","last_change":"2024-03-28T15:50:36.460311+0000","last_active":"2024-03-28T15:51:35.349380+0000","last_peered":"2024-03-28T15:51:35.349380+0000","last_clean":"2024-03-28T15:51:35.349380+0000","last_became_active":"2024-03-28T15:50:36.459934+0000","last_became_peered":"2024-03-28T15:50:36.459934+0000","last_unstale":"2024-03-28T15:51:35.349380+0000","last_undegraded":"2024-03-28T15:51:35.349380+0000","last_fullsized":"2024-03-28T15:51:35.349380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:11.299668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349737+0000","last_change":"2024-03-28T15:50:31.421609+0000","last_active":"2024-03-28T15:51:35.349737+0000","last_peered":"2024-03-28T15:51:35.349737+0000","last_clean":"2024-03-28T15:51:35.349737+0000","last_became_active":"2024-03-28T15:50:31.420979+0000","last_became_peered":"2024-03-28T15:50:31.420979+0000","last_unstale":"2024-03-28T15:51:35.349737+0000","last_undegraded":"2024-03-28T15:51:35.349737+0000","last_fullsized":"2024-03-28T15:51:35.349737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:44:42.974359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419908+0000","last_change":"2024-03-28T15:50:32.432909+0000","last_active":"2024-03-28T15:52:04.419908+0000","last_peered":"2024-03-28T15:52:04.419908+0000","last_clean":"2024-03-28T15:52:04.419908+0000","last_became_active":"2024-03-28T15:50:32.432167+0000","last_became_peered":"2024-03-28T15:50:32.432167+0000","last_unstale":"2024-03-28T15:52:04.419908+0000","last_undegraded":"2024-03-28T15:52:04.419908+0000","last_fullsized":"2024-03-28T15:52:04.419908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:20.168299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349303+0000","last_change":"2024-03-28T15:50:36.459184+0000","last_active":"2024-03-28T15:51:35.349303+0000","last_peered":"2024-03-28T15:51:35.349303+0000","last_clean":"2024-03-28T15:51:35.349303+0000","last_became_active":"2024-03-28T15:50:36.458807+0000","last_became_peered":"2024-03-28T15:50:36.458807+0000","last_unstale":"2024-03-28T15:51:35.349303+0000","last_undegraded":"2024-03-28T15:51:35.349303+0000","last_fullsized":"2024-03-28T15:51:35.349303+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:51.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349302+0000","last_change":"2024-03-28T15:50:31.422344+0000","last_active":"2024-03-28T15:51:35.349302+0000","last_peered":"2024-03-28T15:51:35.349302+0000","last_clean":"2024-03-28T15:51:35.349302+0000","last_became_active":"2024-03-28T15:50:31.421418+0000","last_became_peered":"2024-03-28T15:50:31.421418+0000","last_unstale":"2024-03-28T15:51:35.349302+0000","last_undegraded":"2024-03-28T15:51:35.349302+0000","last_fullsized":"2024-03-28T15:51:35.349302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:17:11.341382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.414157+0000","last_change":"2024-03-28T15:50:32.428599+0000","last_active":"2024-03-28T15:52:04.414157+0000","last_peered":"2024-03-28T15:52:04.414157+0000","last_clean":"2024-03-28T15:52:04.414157+0000","last_became_active":"2024-03-28T15:50:32.427823+0000","last_became_peered":"2024-03-28T15:50:32.427823+0000","last_unstale":"2024-03-28T15:52:04.414157+0000","last_undegraded":"2024-03-28T15:52:04.414157+0000","last_fullsized":"2024-03-28T15:52:04.414157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:18.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351764+0000","last_change":"2024-03-28T15:50:36.451717+0000","last_active":"2024-03-28T15:51:35.351764+0000","last_peered":"2024-03-28T15:51:35.351764+0000","last_clean":"2024-03-28T15:51:35.351764+0000","last_became_active":"2024-03-28T15:50:36.451217+0000","last_became_peered":"2024-03-28T15:50:36.451217+0000","last_unstale":"2024-03-28T15:51:35.351764+0000","last_undegraded":"2024-03-28T15:51:35.351764+0000","last_fullsized":"2024-03-28T15:51:35.351764+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:29:05.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:05.291473+0000","last_change":"2024-03-28T15:50:31.419469+0000","last_active":"2024-03-28T15:52:05.291473+0000","last_peered":"2024-03-28T15:52:05.291473+0000","last_clean":"2024-03-28T15:52:05.291473+0000","last_became_active":"2024-03-28T15:50:31.419162+0000","last_became_peered":"2024-03-28T15:50:31.419162+0000","last_unstale":"2024-03-28T15:52:05.291473+0000","last_undegraded":"2024-03-28T15:52:05.291473+0000","last_fullsized":"2024-03-28T15:52:05.291473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:25.697846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398133+0000","last_change":"2024-03-28T15:50:32.422764+0000","last_active":"2024-03-28T15:52:04.398133+0000","last_peered":"2024-03-28T15:52:04.398133+0000","last_clean":"2024-03-28T15:52:04.398133+0000","last_became_active":"2024-03-28T15:50:32.422368+0000","last_became_peered":"2024-03-28T15:50:32.422368+0000","last_unstale":"2024-03-28T15:52:04.398133+0000","last_undegraded":"2024-03-28T15:52:04.398133+0000","last_fullsized":"2024-03-28T15:52:04.398133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:37:50.195055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352741+0000","last_change":"2024-03-28T15:50:36.453526+0000","last_active":"2024-03-28T15:51:35.352741+0000","last_peered":"2024-03-28T15:51:35.352741+0000","last_clean":"2024-03-28T15:51:35.352741+0000","last_became_active":"2024-03-28T15:50:36.452099+0000","last_became_peered":"2024-03-28T15:50:36.452099+0000","last_unstale":"2024-03-28T15:51:35.352741+0000","last_undegraded":"2024-03-28T15:51:35.352741+0000","last_fullsized":"2024-03-28T15:51:35.352741+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:55:06.379921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.564402+0000","last_change":"2024-03-28T15:50:31.423636+0000","last_active":"2024-03-28T15:51:35.564402+0000","last_peered":"2024-03-28T15:51:35.564402+0000","last_clean":"2024-03-28T15:51:35.564402+0000","last_became_active":"2024-03-28T15:50:31.423303+0000","last_became_peered":"2024-03-28T15:50:31.423303+0000","last_unstale":"2024-03-28T15:51:35.564402+0000","last_undegraded":"2024-03-28T15:51:35.564402+0000","last_fullsized":"2024-03-28T15:51:35.564402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:05:57.666490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410457+0000","last_change":"2024-03-28T15:50:32.428292+0000","last_active":"2024-03-28T15:52:04.410457+0000","last_peered":"2024-03-28T15:52:04.410457+0000","last_clean":"2024-03-28T15:52:04.410457+0000","last_became_active":"2024-03-28T15:50:32.427950+0000","last_became_peered":"2024-03-28T15:50:32.427950+0000","last_unstale":"2024-03-28T15:52:04.410457+0000","last_undegraded":"2024-03-28T15:52:04.410457+0000","last_fullsized":"2024-03-28T15:52:04.410457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:43.040690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353828+0000","last_change":"2024-03-28T15:50:36.461197+0000","last_active":"2024-03-28T15:51:35.353828+0000","last_peered":"2024-03-28T15:51:35.353828+0000","last_clean":"2024-03-28T15:51:35.353828+0000","last_became_active":"2024-03-28T15:50:36.460763+0000","last_became_peered":"2024-03-28T15:50:36.460763+0000","last_unstale":"2024-03-28T15:51:35.353828+0000","last_undegraded":"2024-03-28T15:51:35.353828+0000","last_fullsized":"2024-03-28T15:51:35.353828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:23.500309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934556+0000","last_change":"2024-03-28T15:50:31.417000+0000","last_active":"2024-03-28T15:51:35.934556+0000","last_peered":"2024-03-28T15:51:35.934556+0000","last_clean":"2024-03-28T15:51:35.934556+0000","last_became_active":"2024-03-28T15:50:31.416652+0000","last_became_peered":"2024-03-28T15:50:31.416652+0000","last_unstale":"2024-03-28T15:51:35.934556+0000","last_undegraded":"2024-03-28T15:51:35.934556+0000","last_fullsized":"2024-03-28T15:51:35.934556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:06:55.338636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409999+0000","last_change":"2024-03-28T15:51:35.936192+0000","last_active":"2024-03-28T15:52:04.409999+0000","last_peered":"2024-03-28T15:52:04.409999+0000","last_clean":"2024-03-28T15:52:04.409999+0000","last_became_active":"2024-03-28T15:51:35.935863+0000","last_became_peered":"2024-03-28T15:51:35.935863+0000","last_unstale":"2024-03-28T15:52:04.409999+0000","last_undegraded":"2024-03-28T15:52:04.409999+0000","last_fullsized":"2024-03-28T15:52:04.409999+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:30:38.502087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353387+0000","last_change":"2024-03-28T15:50:36.460876+0000","last_active":"2024-03-28T15:51:35.353387+0000","last_peered":"2024-03-28T15:51:35.353387+0000","last_clean":"2024-03-28T15:51:35.353387+0000","last_became_active":"2024-03-28T15:50:36.460452+0000","last_became_peered":"2024-03-28T15:50:36.460452+0000","last_unstale":"2024-03-28T15:51:35.353387+0000","last_undegraded":"2024-03-28T15:51:35.353387+0000","last_fullsized":"2024-03-28T15:51:35.353387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:08:36.260861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351897+0000","last_change":"2024-03-28T15:50:31.417925+0000","last_active":"2024-03-28T15:51:35.351897+0000","last_peered":"2024-03-28T15:51:35.351897+0000","last_clean":"2024-03-28T15:51:35.351897+0000","last_became_active":"2024-03-28T15:50:31.417615+0000","last_became_peered":"2024-03-28T15:50:31.417615+0000","last_unstale":"2024-03-28T15:51:35.351897+0000","last_undegraded":"2024-03-28T15:51:35.351897+0000","last_fullsized":"2024-03-28T15:51:35.351897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:31.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408474+0000","last_change":"2024-03-28T15:50:32.423511+0000","last_active":"2024-03-28T15:52:04.408474+0000","last_peered":"2024-03-28T15:52:04.408474+0000","last_clean":"2024-03-28T15:52:04.408474+0000","last_became_active":"2024-03-28T15:50:32.423194+0000","last_became_peered":"2024-03-28T15:50:32.423194+0000","last_unstale":"2024-03-28T15:52:04.408474+0000","last_undegraded":"2024-03-28T15:52:04.408474+0000","last_fullsized":"2024-03-28T15:52:04.408474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:44.170488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.853350+0000","last_change":"2024-03-28T15:50:36.465051+0000","last_active":"2024-03-28T15:51:35.853350+0000","last_peered":"2024-03-28T15:51:35.853350+0000","last_clean":"2024-03-28T15:51:35.853350+0000","last_became_active":"2024-03-28T15:50:36.464579+0000","last_became_peered":"2024-03-28T15:50:36.464579+0000","last_unstale":"2024-03-28T15:51:35.853350+0000","last_undegraded":"2024-03-28T15:51:35.853350+0000","last_fullsized":"2024-03-28T15:51:35.853350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:56.769485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":174,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:09.294631+0000","last_change":"2024-03-28T15:50:31.429484+0000","last_active":"2024-03-28T15:52:09.294631+0000","last_peered":"2024-03-28T15:52:09.294631+0000","last_clean":"2024-03-28T15:52:09.294631+0000","last_became_active":"2024-03-28T15:50:31.429206+0000","last_became_peered":"2024-03-28T15:50:31.429206+0000","last_unstale":"2024-03-28T15:52:09.294631+0000","last_undegraded":"2024-03-28T15:52:09.294631+0000","last_fullsized":"2024-03-28T15:52:09.294631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:05.599043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419336+0000","last_change":"2024-03-28T15:50:32.421813+0000","last_active":"2024-03-28T15:52:04.419336+0000","last_peered":"2024-03-28T15:52:04.419336+0000","last_clean":"2024-03-28T15:52:04.419336+0000","last_became_active":"2024-03-28T15:50:32.421510+0000","last_became_peered":"2024-03-28T15:50:32.421510+0000","last_unstale":"2024-03-28T15:52:04.419336+0000","last_undegraded":"2024-03-28T15:52:04.419336+0000","last_fullsized":"2024-03-28T15:52:04.419336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:06.314487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.348840+0000","last_change":"2024-03-28T15:50:36.464148+0000","last_active":"2024-03-28T15:51:35.348840+0000","last_peered":"2024-03-28T15:51:35.348840+0000","last_clean":"2024-03-28T15:51:35.348840+0000","last_became_active":"2024-03-28T15:50:36.463696+0000","last_became_peered":"2024-03-28T15:50:36.463696+0000","last_unstale":"2024-03-28T15:51:35.348840+0000","last_undegraded":"2024-03-28T15:51:35.348840+0000","last_fullsized":"2024-03-28T15:51:35.348840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:57.733727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353228+0000","last_change":"2024-03-28T15:50:31.422969+0000","last_active":"2024-03-28T15:51:35.353228+0000","last_peered":"2024-03-28T15:51:35.353228+0000","last_clean":"2024-03-28T15:51:35.353228+0000","last_became_active":"2024-03-28T15:50:31.422664+0000","last_became_peered":"2024-03-28T15:50:31.422664+0000","last_unstale":"2024-03-28T15:51:35.353228+0000","last_undegraded":"2024-03-28T15:51:35.353228+0000","last_fullsized":"2024-03-28T15:51:35.353228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:24.469105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407536+0000","last_change":"2024-03-28T15:51:35.357854+0000","last_active":"2024-03-28T15:52:04.407536+0000","last_peered":"2024-03-28T15:52:04.407536+0000","last_clean":"2024-03-28T15:52:04.407536+0000","last_became_active":"2024-03-28T15:51:35.357471+0000","last_became_peered":"2024-03-28T15:51:35.357471+0000","last_unstale":"2024-03-28T15:52:04.407536+0000","last_undegraded":"2024-03-28T15:52:04.407536+0000","last_fullsized":"2024-03-28T15:52:04.407536+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:58.970860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349655+0000","last_change":"2024-03-28T15:50:36.457721+0000","last_active":"2024-03-28T15:51:35.349655+0000","last_peered":"2024-03-28T15:51:35.349655+0000","last_clean":"2024-03-28T15:51:35.349655+0000","last_became_active":"2024-03-28T15:50:36.457028+0000","last_became_peered":"2024-03-28T15:50:36.457028+0000","last_unstale":"2024-03-28T15:51:35.349655+0000","last_undegraded":"2024-03-28T15:51:35.349655+0000","last_fullsized":"2024-03-28T15:51:35.349655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:47.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352729+0000","last_change":"2024-03-28T15:50:31.418265+0000","last_active":"2024-03-28T15:51:35.352729+0000","last_peered":"2024-03-28T15:51:35.352729+0000","last_clean":"2024-03-28T15:51:35.352729+0000","last_became_active":"2024-03-28T15:50:31.417944+0000","last_became_peered":"2024-03-28T15:50:31.417944+0000","last_unstale":"2024-03-28T15:51:35.352729+0000","last_undegraded":"2024-03-28T15:51:35.352729+0000","last_fullsized":"2024-03-28T15:51:35.352729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:58.633512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.421435+0000","last_change":"2024-03-28T15:50:32.433180+0000","last_active":"2024-03-28T15:52:04.421435+0000","last_peered":"2024-03-28T15:52:04.421435+0000","last_clean":"2024-03-28T15:52:04.421435+0000","last_became_active":"2024-03-28T15:50:32.432598+0000","last_became_peered":"2024-03-28T15:50:32.432598+0000","last_unstale":"2024-03-28T15:52:04.421435+0000","last_undegraded":"2024-03-28T15:52:04.421435+0000","last_fullsized":"2024-03-28T15:52:04.421435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:16.051480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.348851+0000","last_change":"2024-03-28T15:50:36.453471+0000","last_active":"2024-03-28T15:51:35.348851+0000","last_peered":"2024-03-28T15:51:35.348851+0000","last_clean":"2024-03-28T15:51:35.348851+0000","last_became_active":"2024-03-28T15:50:36.453070+0000","last_became_peered":"2024-03-28T15:50:36.453070+0000","last_unstale":"2024-03-28T15:51:35.348851+0000","last_undegraded":"2024-03-28T15:51:35.348851+0000","last_fullsized":"2024-03-28T15:51:35.348851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:55.417086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:09.296397+0000","last_change":"2024-03-28T15:50:31.419585+0000","last_active":"2024-03-28T15:52:09.296397+0000","last_peered":"2024-03-28T15:52:09.296397+0000","last_clean":"2024-03-28T15:52:09.296397+0000","last_became_active":"2024-03-28T15:50:31.419241+0000","last_became_peered":"2024-03-28T15:50:31.419241+0000","last_unstale":"2024-03-28T15:52:09.296397+0000","last_undegraded":"2024-03-28T15:52:09.296397+0000","last_fullsized":"2024-03-28T15:52:09.296397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:34:17.438115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59120,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":150,"num_read_kb":133,"num_write":38,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409196+0000","last_change":"2024-03-28T15:50:32.436526+0000","last_active":"2024-03-28T15:52:04.409196+0000","last_peered":"2024-03-28T15:52:04.409196+0000","last_clean":"2024-03-28T15:52:04.409196+0000","last_became_active":"2024-03-28T15:50:32.436211+0000","last_became_peered":"2024-03-28T15:50:32.436211+0000","last_unstale":"2024-03-28T15:52:04.409196+0000","last_undegraded":"2024-03-28T15:52:04.409196+0000","last_fullsized":"2024-03-28T15:52:04.409196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:04.424281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353122+0000","last_change":"2024-03-28T15:50:36.460308+0000","last_active":"2024-03-28T15:51:35.353122+0000","last_peered":"2024-03-28T15:51:35.353122+0000","last_clean":"2024-03-28T15:51:35.353122+0000","last_became_active":"2024-03-28T15:50:36.459417+0000","last_became_peered":"2024-03-28T15:50:36.459417+0000","last_unstale":"2024-03-28T15:51:35.353122+0000","last_undegraded":"2024-03-28T15:51:35.353122+0000","last_fullsized":"2024-03-28T15:51:35.353122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:22.022249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351574+0000","last_change":"2024-03-28T15:50:31.426829+0000","last_active":"2024-03-28T15:51:35.351574+0000","last_peered":"2024-03-28T15:51:35.351574+0000","last_clean":"2024-03-28T15:51:35.351574+0000","last_became_active":"2024-03-28T15:50:31.426434+0000","last_became_peered":"2024-03-28T15:50:31.426434+0000","last_unstale":"2024-03-28T15:51:35.351574+0000","last_undegraded":"2024-03-28T15:51:35.351574+0000","last_fullsized":"2024-03-28T15:51:35.351574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.567766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.404179+0000","last_change":"2024-03-28T15:50:32.427927+0000","last_active":"2024-03-28T15:52:04.404179+0000","last_peered":"2024-03-28T15:52:04.404179+0000","last_clean":"2024-03-28T15:52:04.404179+0000","last_became_active":"2024-03-28T15:50:32.427609+0000","last_became_peered":"2024-03-28T15:50:32.427609+0000","last_unstale":"2024-03-28T15:52:04.404179+0000","last_undegraded":"2024-03-28T15:52:04.404179+0000","last_fullsized":"2024-03-28T15:52:04.404179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:49.120638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351821+0000","last_change":"2024-03-28T15:50:36.460355+0000","last_active":"2024-03-28T15:51:35.351821+0000","last_peered":"2024-03-28T15:51:35.351821+0000","last_clean":"2024-03-28T15:51:35.351821+0000","last_became_active":"2024-03-28T15:50:36.459935+0000","last_became_peered":"2024-03-28T15:50:36.459935+0000","last_unstale":"2024-03-28T15:51:35.351821+0000","last_undegraded":"2024-03-28T15:51:35.351821+0000","last_fullsized":"2024-03-28T15:51:35.351821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.895844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352405+0000","last_change":"2024-03-28T15:50:31.417282+0000","last_active":"2024-03-28T15:51:35.352405+0000","last_peered":"2024-03-28T15:51:35.352405+0000","last_clean":"2024-03-28T15:51:35.352405+0000","last_became_active":"2024-03-28T15:50:31.416852+0000","last_became_peered":"2024-03-28T15:50:31.416852+0000","last_unstale":"2024-03-28T15:51:35.352405+0000","last_undegraded":"2024-03-28T15:51:35.352405+0000","last_fullsized":"2024-03-28T15:51:35.352405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:26:12.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410320+0000","last_change":"2024-03-28T15:50:32.430362+0000","last_active":"2024-03-28T15:52:04.410320+0000","last_peered":"2024-03-28T15:52:04.410320+0000","last_clean":"2024-03-28T15:52:04.410320+0000","last_became_active":"2024-03-28T15:50:32.429999+0000","last_became_peered":"2024-03-28T15:50:32.429999+0000","last_unstale":"2024-03-28T15:52:04.410320+0000","last_undegraded":"2024-03-28T15:52:04.410320+0000","last_fullsized":"2024-03-28T15:52:04.410320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:47.613322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349849+0000","last_change":"2024-03-28T15:50:36.463116+0000","last_active":"2024-03-28T15:51:35.349849+0000","last_peered":"2024-03-28T15:51:35.349849+0000","last_clean":"2024-03-28T15:51:35.349849+0000","last_became_active":"2024-03-28T15:50:36.462705+0000","last_became_peered":"2024-03-28T15:50:36.462705+0000","last_unstale":"2024-03-28T15:51:35.349849+0000","last_undegraded":"2024-03-28T15:51:35.349849+0000","last_fullsized":"2024-03-28T15:51:35.349849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:49.876404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:06.300931+0000","last_change":"2024-03-28T15:50:31.418595+0000","last_active":"2024-03-28T15:52:06.300931+0000","last_peered":"2024-03-28T15:52:06.300931+0000","last_clean":"2024-03-28T15:52:06.300931+0000","last_became_active":"2024-03-28T15:50:31.418185+0000","last_became_peered":"2024-03-28T15:50:31.418185+0000","last_unstale":"2024-03-28T15:52:06.300931+0000","last_undegraded":"2024-03-28T15:52:06.300931+0000","last_fullsized":"2024-03-28T15:52:06.300931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:40.246760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407733+0000","last_change":"2024-03-28T15:50:32.431867+0000","last_active":"2024-03-28T15:52:04.407733+0000","last_peered":"2024-03-28T15:52:04.407733+0000","last_clean":"2024-03-28T15:52:04.407733+0000","last_became_active":"2024-03-28T15:50:32.431552+0000","last_became_peered":"2024-03-28T15:50:32.431552+0000","last_unstale":"2024-03-28T15:52:04.407733+0000","last_undegraded":"2024-03-28T15:52:04.407733+0000","last_fullsized":"2024-03-28T15:52:04.407733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:20.849087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.935008+0000","last_change":"2024-03-28T15:50:36.466785+0000","last_active":"2024-03-28T15:51:35.935008+0000","last_peered":"2024-03-28T15:51:35.935008+0000","last_clean":"2024-03-28T15:51:35.935008+0000","last_became_active":"2024-03-28T15:50:36.466356+0000","last_became_peered":"2024-03-28T15:50:36.466356+0000","last_unstale":"2024-03-28T15:51:35.935008+0000","last_undegraded":"2024-03-28T15:51:35.935008+0000","last_fullsized":"2024-03-28T15:51:35.935008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:07.453802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349090+0000","last_change":"2024-03-28T15:50:31.417956+0000","last_active":"2024-03-28T15:51:35.349090+0000","last_peered":"2024-03-28T15:51:35.349090+0000","last_clean":"2024-03-28T15:51:35.349090+0000","last_became_active":"2024-03-28T15:50:31.417613+0000","last_became_peered":"2024-03-28T15:50:31.417613+0000","last_unstale":"2024-03-28T15:51:35.349090+0000","last_undegraded":"2024-03-28T15:51:35.349090+0000","last_fullsized":"2024-03-28T15:51:35.349090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:33.039685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.413414+0000","last_change":"2024-03-28T15:51:35.936715+0000","last_active":"2024-03-28T15:52:04.413414+0000","last_peered":"2024-03-28T15:52:04.413414+0000","last_clean":"2024-03-28T15:52:04.413414+0000","last_became_active":"2024-03-28T15:51:35.936451+0000","last_became_peered":"2024-03-28T15:51:35.936451+0000","last_unstale":"2024-03-28T15:52:04.413414+0000","last_undegraded":"2024-03-28T15:52:04.413414+0000","last_fullsized":"2024-03-28T15:52:04.413414+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:50.654204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352550+0000","last_change":"2024-03-28T15:50:36.463509+0000","last_active":"2024-03-28T15:51:35.352550+0000","last_peered":"2024-03-28T15:51:35.352550+0000","last_clean":"2024-03-28T15:51:35.352550+0000","last_became_active":"2024-03-28T15:50:36.462733+0000","last_became_peered":"2024-03-28T15:50:36.462733+0000","last_unstale":"2024-03-28T15:51:35.352550+0000","last_undegraded":"2024-03-28T15:51:35.352550+0000","last_fullsized":"2024-03-28T15:51:35.352550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:43:33.737012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349865+0000","last_change":"2024-03-28T15:50:31.420224+0000","last_active":"2024-03-28T15:51:35.349865+0000","last_peered":"2024-03-28T15:51:35.349865+0000","last_clean":"2024-03-28T15:51:35.349865+0000","last_became_active":"2024-03-28T15:50:31.419913+0000","last_became_peered":"2024-03-28T15:50:31.419913+0000","last_unstale":"2024-03-28T15:51:35.349865+0000","last_undegraded":"2024-03-28T15:51:35.349865+0000","last_fullsized":"2024-03-28T15:51:35.349865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:59:49.834297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406280+0000","last_change":"2024-03-28T15:51:35.357722+0000","last_active":"2024-03-28T15:52:04.406280+0000","last_peered":"2024-03-28T15:52:04.406280+0000","last_clean":"2024-03-28T15:52:04.406280+0000","last_became_active":"2024-03-28T15:51:35.357352+0000","last_became_peered":"2024-03-28T15:51:35.357352+0000","last_unstale":"2024-03-28T15:52:04.406280+0000","last_undegraded":"2024-03-28T15:52:04.406280+0000","last_fullsized":"2024-03-28T15:52:04.406280+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:13.934522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349311+0000","last_change":"2024-03-28T15:50:36.460744+0000","last_active":"2024-03-28T15:51:35.349311+0000","last_peered":"2024-03-28T15:51:35.349311+0000","last_clean":"2024-03-28T15:51:35.349311+0000","last_became_active":"2024-03-28T15:50:36.460276+0000","last_became_peered":"2024-03-28T15:50:36.460276+0000","last_unstale":"2024-03-28T15:51:35.349311+0000","last_undegraded":"2024-03-28T15:51:35.349311+0000","last_fullsized":"2024-03-28T15:51:35.349311+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:12.600154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349390+0000","last_change":"2024-03-28T15:50:31.427718+0000","last_active":"2024-03-28T15:51:35.349390+0000","last_peered":"2024-03-28T15:51:35.349390+0000","last_clean":"2024-03-28T15:51:35.349390+0000","last_became_active":"2024-03-28T15:50:31.427414+0000","last_became_peered":"2024-03-28T15:50:31.427414+0000","last_unstale":"2024-03-28T15:51:35.349390+0000","last_undegraded":"2024-03-28T15:51:35.349390+0000","last_fullsized":"2024-03-28T15:51:35.349390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:24.636431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409642+0000","last_change":"2024-03-28T15:50:32.424409+0000","last_active":"2024-03-28T15:52:04.409642+0000","last_peered":"2024-03-28T15:52:04.409642+0000","last_clean":"2024-03-28T15:52:04.409642+0000","last_became_active":"2024-03-28T15:50:32.424108+0000","last_became_peered":"2024-03-28T15:50:32.424108+0000","last_unstale":"2024-03-28T15:52:04.409642+0000","last_undegraded":"2024-03-28T15:52:04.409642+0000","last_fullsized":"2024-03-28T15:52:04.409642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:59:23.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934923+0000","last_change":"2024-03-28T15:50:36.463533+0000","last_active":"2024-03-28T15:51:35.934923+0000","last_peered":"2024-03-28T15:51:35.934923+0000","last_clean":"2024-03-28T15:51:35.934923+0000","last_became_active":"2024-03-28T15:50:36.463047+0000","last_became_peered":"2024-03-28T15:50:36.463047+0000","last_unstale":"2024-03-28T15:51:35.934923+0000","last_undegraded":"2024-03-28T15:51:35.934923+0000","last_fullsized":"2024-03-28T15:51:35.934923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:52.072250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:05.299814+0000","last_change":"2024-03-28T15:50:31.417073+0000","last_active":"2024-03-28T15:52:05.299814+0000","last_peered":"2024-03-28T15:52:05.299814+0000","last_clean":"2024-03-28T15:52:05.299814+0000","last_became_active":"2024-03-28T15:50:31.416755+0000","last_became_peered":"2024-03-28T15:50:31.416755+0000","last_unstale":"2024-03-28T15:52:05.299814+0000","last_undegraded":"2024-03-28T15:52:05.299814+0000","last_fullsized":"2024-03-28T15:52:05.299814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:45.887894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398987+0000","last_change":"2024-03-28T15:50:32.424912+0000","last_active":"2024-03-28T15:52:04.398987+0000","last_peered":"2024-03-28T15:52:04.398987+0000","last_clean":"2024-03-28T15:52:04.398987+0000","last_became_active":"2024-03-28T15:50:32.424609+0000","last_became_peered":"2024-03-28T15:50:32.424609+0000","last_unstale":"2024-03-28T15:52:04.398987+0000","last_undegraded":"2024-03-28T15:52:04.398987+0000","last_fullsized":"2024-03-28T15:52:04.398987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:37.539793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934909+0000","last_change":"2024-03-28T15:50:36.463968+0000","last_active":"2024-03-28T15:51:35.934909+0000","last_peered":"2024-03-28T15:51:35.934909+0000","last_clean":"2024-03-28T15:51:35.934909+0000","last_became_active":"2024-03-28T15:50:36.460697+0000","last_became_peered":"2024-03-28T15:50:36.460697+0000","last_unstale":"2024-03-28T15:51:35.934909+0000","last_undegraded":"2024-03-28T15:51:35.934909+0000","last_fullsized":"2024-03-28T15:51:35.934909+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:00.726769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403083+0000","last_change":"2024-03-28T15:50:32.419783+0000","last_active":"2024-03-28T15:52:04.403083+0000","last_peered":"2024-03-28T15:52:04.403083+0000","last_clean":"2024-03-28T15:52:04.403083+0000","last_became_active":"2024-03-28T15:50:32.419404+0000","last_became_peered":"2024-03-28T15:50:32.419404+0000","last_unstale":"2024-03-28T15:52:04.403083+0000","last_undegraded":"2024-03-28T15:52:04.403083+0000","last_fullsized":"2024-03-28T15:52:04.403083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:42.437830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":101,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:06.300454+0000","last_change":"2024-03-28T15:50:31.414630+0000","last_active":"2024-03-28T15:52:06.300454+0000","last_peered":"2024-03-28T15:52:06.300454+0000","last_clean":"2024-03-28T15:52:06.300454+0000","last_became_active":"2024-03-28T15:50:31.414300+0000","last_became_peered":"2024-03-28T15:50:31.414300+0000","last_unstale":"2024-03-28T15:52:06.300454+0000","last_undegraded":"2024-03-28T15:52:06.300454+0000","last_fullsized":"2024-03-28T15:52:06.300454+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:48.773649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353277+0000","last_change":"2024-03-28T15:50:36.462352+0000","last_active":"2024-03-28T15:51:35.353277+0000","last_peered":"2024-03-28T15:51:35.353277+0000","last_clean":"2024-03-28T15:51:35.353277+0000","last_became_active":"2024-03-28T15:50:36.462003+0000","last_became_peered":"2024-03-28T15:50:36.462003+0000","last_unstale":"2024-03-28T15:51:35.353277+0000","last_undegraded":"2024-03-28T15:51:35.353277+0000","last_fullsized":"2024-03-28T15:51:35.353277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:42.830355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408134+0000","last_change":"2024-03-28T15:50:32.429086+0000","last_active":"2024-03-28T15:52:04.408134+0000","last_peered":"2024-03-28T15:52:04.408134+0000","last_clean":"2024-03-28T15:52:04.408134+0000","last_became_active":"2024-03-28T15:50:32.427761+0000","last_became_peered":"2024-03-28T15:50:32.427761+0000","last_unstale":"2024-03-28T15:52:04.408134+0000","last_undegraded":"2024-03-28T15:52:04.408134+0000","last_fullsized":"2024-03-28T15:52:04.408134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.883733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349768+0000","last_change":"2024-03-28T15:50:31.417017+0000","last_active":"2024-03-28T15:51:35.349768+0000","last_peered":"2024-03-28T15:51:35.349768+0000","last_clean":"2024-03-28T15:51:35.349768+0000","last_became_active":"2024-03-28T15:50:31.416688+0000","last_became_peered":"2024-03-28T15:50:31.416688+0000","last_unstale":"2024-03-28T15:51:35.349768+0000","last_undegraded":"2024-03-28T15:51:35.349768+0000","last_fullsized":"2024-03-28T15:51:35.349768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:39.364544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352331+0000","last_change":"2024-03-28T15:49:22.209051+0000","last_active":"2024-03-28T15:51:35.352331+0000","last_peered":"2024-03-28T15:51:35.352331+0000","last_clean":"2024-03-28T15:51:35.352331+0000","last_became_active":"2024-03-28T15:49:22.208633+0000","last_became_peered":"2024-03-28T15:49:22.208633+0000","last_unstale":"2024-03-28T15:51:35.352331+0000","last_undegraded":"2024-03-28T15:51:35.352331+0000","last_fullsized":"2024-03-28T15:51:35.352331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_clean_scrub_stamp":"2024-03-28T15:45:59.266253+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:08:41.194614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351722+0000","last_change":"2024-03-28T15:50:36.463514+0000","last_active":"2024-03-28T15:51:35.351722+0000","last_peered":"2024-03-28T15:51:35.351722+0000","last_clean":"2024-03-28T15:51:35.351722+0000","last_became_active":"2024-03-28T15:50:36.463110+0000","last_became_peered":"2024-03-28T15:50:36.463110+0000","last_unstale":"2024-03-28T15:51:35.351722+0000","last_undegraded":"2024-03-28T15:51:35.351722+0000","last_fullsized":"2024-03-28T15:51:35.351722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:30.481123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402429+0000","last_change":"2024-03-28T15:50:32.427016+0000","last_active":"2024-03-28T15:52:04.402429+0000","last_peered":"2024-03-28T15:52:04.402429+0000","last_clean":"2024-03-28T15:52:04.402429+0000","last_became_active":"2024-03-28T15:50:32.426519+0000","last_became_peered":"2024-03-28T15:50:32.426519+0000","last_unstale":"2024-03-28T15:52:04.402429+0000","last_undegraded":"2024-03-28T15:52:04.402429+0000","last_fullsized":"2024-03-28T15:52:04.402429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:27.650290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352455+0000","last_change":"2024-03-28T15:50:31.420871+0000","last_active":"2024-03-28T15:51:35.352455+0000","last_peered":"2024-03-28T15:51:35.352455+0000","last_clean":"2024-03-28T15:51:35.352455+0000","last_became_active":"2024-03-28T15:50:31.420418+0000","last_became_peered":"2024-03-28T15:50:31.420418+0000","last_unstale":"2024-03-28T15:51:35.352455+0000","last_undegraded":"2024-03-28T15:51:35.352455+0000","last_fullsized":"2024-03-28T15:51:35.352455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:57.615539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352424+0000","last_change":"2024-03-28T15:50:36.457759+0000","last_active":"2024-03-28T15:51:35.352424+0000","last_peered":"2024-03-28T15:51:35.352424+0000","last_clean":"2024-03-28T15:51:35.352424+0000","last_became_active":"2024-03-28T15:50:36.457248+0000","last_became_peered":"2024-03-28T15:50:36.457248+0000","last_unstale":"2024-03-28T15:51:35.352424+0000","last_undegraded":"2024-03-28T15:51:35.352424+0000","last_fullsized":"2024-03-28T15:51:35.352424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:13:30.420180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349512+0000","last_change":"2024-03-28T15:50:31.428270+0000","last_active":"2024-03-28T15:51:35.349512+0000","last_peered":"2024-03-28T15:51:35.349512+0000","last_clean":"2024-03-28T15:51:35.349512+0000","last_became_active":"2024-03-28T15:50:31.427888+0000","last_became_peered":"2024-03-28T15:50:31.427888+0000","last_unstale":"2024-03-28T15:51:35.349512+0000","last_undegraded":"2024-03-28T15:51:35.349512+0000","last_fullsized":"2024-03-28T15:51:35.349512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:48:51.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.401429+0000","last_change":"2024-03-28T15:50:32.416631+0000","last_active":"2024-03-28T15:52:04.401429+0000","last_peered":"2024-03-28T15:52:04.401429+0000","last_clean":"2024-03-28T15:52:04.401429+0000","last_became_active":"2024-03-28T15:50:32.416300+0000","last_became_peered":"2024-03-28T15:50:32.416300+0000","last_unstale":"2024-03-28T15:52:04.401429+0000","last_undegraded":"2024-03-28T15:52:04.401429+0000","last_fullsized":"2024-03-28T15:52:04.401429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:19.158562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.853207+0000","last_change":"2024-03-28T15:50:36.465991+0000","last_active":"2024-03-28T15:51:35.853207+0000","last_peered":"2024-03-28T15:51:35.853207+0000","last_clean":"2024-03-28T15:51:35.853207+0000","last_became_active":"2024-03-28T15:50:36.465506+0000","last_became_peered":"2024-03-28T15:50:36.465506+0000","last_unstale":"2024-03-28T15:51:35.853207+0000","last_undegraded":"2024-03-28T15:51:35.853207+0000","last_fullsized":"2024-03-28T15:51:35.853207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:22.889068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349979+0000","last_change":"2024-03-28T15:50:31.423894+0000","last_active":"2024-03-28T15:51:35.349979+0000","last_peered":"2024-03-28T15:51:35.349979+0000","last_clean":"2024-03-28T15:51:35.349979+0000","last_became_active":"2024-03-28T15:50:31.423562+0000","last_became_peered":"2024-03-28T15:50:31.423562+0000","last_unstale":"2024-03-28T15:51:35.349979+0000","last_undegraded":"2024-03-28T15:51:35.349979+0000","last_fullsized":"2024-03-28T15:51:35.349979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:55.065632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409951+0000","last_change":"2024-03-28T15:50:32.435132+0000","last_active":"2024-03-28T15:52:04.409951+0000","last_peered":"2024-03-28T15:52:04.409951+0000","last_clean":"2024-03-28T15:52:04.409951+0000","last_became_active":"2024-03-28T15:50:32.434523+0000","last_became_peered":"2024-03-28T15:50:32.434523+0000","last_unstale":"2024-03-28T15:52:04.409951+0000","last_undegraded":"2024-03-28T15:52:04.409951+0000","last_fullsized":"2024-03-28T15:52:04.409951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:35.363388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351262+0000","last_change":"2024-03-28T15:50:36.453501+0000","last_active":"2024-03-28T15:51:35.351262+0000","last_peered":"2024-03-28T15:51:35.351262+0000","last_clean":"2024-03-28T15:51:35.351262+0000","last_became_active":"2024-03-28T15:50:36.452121+0000","last_became_peered":"2024-03-28T15:50:36.452121+0000","last_unstale":"2024-03-28T15:51:35.351262+0000","last_undegraded":"2024-03-28T15:51:35.351262+0000","last_fullsized":"2024-03-28T15:51:35.351262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:48:33.354233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.350253+0000","last_change":"2024-03-28T15:50:31.425645+0000","last_active":"2024-03-28T15:51:35.350253+0000","last_peered":"2024-03-28T15:51:35.350253+0000","last_clean":"2024-03-28T15:51:35.350253+0000","last_became_active":"2024-03-28T15:50:31.425332+0000","last_became_peered":"2024-03-28T15:50:31.425332+0000","last_unstale":"2024-03-28T15:51:35.350253+0000","last_undegraded":"2024-03-28T15:51:35.350253+0000","last_fullsized":"2024-03-28T15:51:35.350253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:16.221936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410597+0000","last_change":"2024-03-28T15:50:32.432602+0000","last_active":"2024-03-28T15:52:04.410597+0000","last_peered":"2024-03-28T15:52:04.410597+0000","last_clean":"2024-03-28T15:52:04.410597+0000","last_became_active":"2024-03-28T15:50:32.432303+0000","last_became_peered":"2024-03-28T15:50:32.432303+0000","last_unstale":"2024-03-28T15:52:04.410597+0000","last_undegraded":"2024-03-28T15:52:04.410597+0000","last_fullsized":"2024-03-28T15:52:04.410597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.524582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.566352+0000","last_change":"2024-03-28T15:51:35.566352+0000","last_active":"2024-03-28T15:51:35.566352+0000","last_peered":"2024-03-28T15:51:35.566352+0000","last_clean":"2024-03-28T15:51:35.566352+0000","last_became_active":"2024-03-28T15:51:35.565829+0000","last_became_peered":"2024-03-28T15:51:35.565829+0000","last_unstale":"2024-03-28T15:51:35.566352+0000","last_undegraded":"2024-03-28T15:51:35.566352+0000","last_fullsized":"2024-03-28T15:51:35.566352+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:00:19.067884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.718257+0000","last_change":"2024-03-28T15:50:31.418403+0000","last_active":"2024-03-28T15:51:35.718257+0000","last_peered":"2024-03-28T15:51:35.718257+0000","last_clean":"2024-03-28T15:51:35.718257+0000","last_became_active":"2024-03-28T15:50:31.418102+0000","last_became_peered":"2024-03-28T15:50:31.418102+0000","last_unstale":"2024-03-28T15:51:35.718257+0000","last_undegraded":"2024-03-28T15:51:35.718257+0000","last_fullsized":"2024-03-28T15:51:35.718257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:01:56.175404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.370732+0000","last_change":"2024-03-28T15:50:32.417331+0000","last_active":"2024-03-28T15:52:04.370732+0000","last_peered":"2024-03-28T15:52:04.370732+0000","last_clean":"2024-03-28T15:52:04.370732+0000","last_became_active":"2024-03-28T15:50:32.417026+0000","last_became_peered":"2024-03-28T15:50:32.417026+0000","last_unstale":"2024-03-28T15:52:04.370732+0000","last_undegraded":"2024-03-28T15:52:04.370732+0000","last_fullsized":"2024-03-28T15:52:04.370732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:24.081575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349990+0000","last_change":"2024-03-28T15:50:36.454877+0000","last_active":"2024-03-28T15:51:35.349990+0000","last_peered":"2024-03-28T15:51:35.349990+0000","last_clean":"2024-03-28T15:51:35.349990+0000","last_became_active":"2024-03-28T15:50:36.453802+0000","last_became_peered":"2024-03-28T15:50:36.453802+0000","last_unstale":"2024-03-28T15:51:35.349990+0000","last_undegraded":"2024-03-28T15:51:35.349990+0000","last_fullsized":"2024-03-28T15:51:35.349990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:56.479558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.852685+0000","last_change":"2024-03-28T15:50:31.425672+0000","last_active":"2024-03-28T15:51:35.852685+0000","last_peered":"2024-03-28T15:51:35.852685+0000","last_clean":"2024-03-28T15:51:35.852685+0000","last_became_active":"2024-03-28T15:50:31.425324+0000","last_became_peered":"2024-03-28T15:50:31.425324+0000","last_unstale":"2024-03-28T15:51:35.852685+0000","last_undegraded":"2024-03-28T15:51:35.852685+0000","last_fullsized":"2024-03-28T15:51:35.852685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:48:46.380962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408297+0000","last_change":"2024-03-28T15:50:32.424831+0000","last_active":"2024-03-28T15:52:04.408297+0000","last_peered":"2024-03-28T15:52:04.408297+0000","last_clean":"2024-03-28T15:52:04.408297+0000","last_became_active":"2024-03-28T15:50:32.423678+0000","last_became_peered":"2024-03-28T15:50:32.423678+0000","last_unstale":"2024-03-28T15:52:04.408297+0000","last_undegraded":"2024-03-28T15:52:04.408297+0000","last_fullsized":"2024-03-28T15:52:04.408297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:35:06.895634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.570333+0000","last_change":"2024-03-28T15:51:35.570333+0000","last_active":"2024-03-28T15:51:35.570333+0000","last_peered":"2024-03-28T15:51:35.570333+0000","last_clean":"2024-03-28T15:51:35.570333+0000","last_became_active":"2024-03-28T15:51:35.566494+0000","last_became_peered":"2024-03-28T15:51:35.566494+0000","last_unstale":"2024-03-28T15:51:35.570333+0000","last_undegraded":"2024-03-28T15:51:35.570333+0000","last_fullsized":"2024-03-28T15:51:35.570333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:47:34.617415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351259+0000","last_change":"2024-03-28T15:50:31.430248+0000","last_active":"2024-03-28T15:51:35.351259+0000","last_peered":"2024-03-28T15:51:35.351259+0000","last_clean":"2024-03-28T15:51:35.351259+0000","last_became_active":"2024-03-28T15:50:31.429964+0000","last_became_peered":"2024-03-28T15:50:31.429964+0000","last_unstale":"2024-03-28T15:51:35.351259+0000","last_undegraded":"2024-03-28T15:51:35.351259+0000","last_fullsized":"2024-03-28T15:51:35.351259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:10:25.193102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408504+0000","last_change":"2024-03-28T15:50:32.431687+0000","last_active":"2024-03-28T15:52:04.408504+0000","last_peered":"2024-03-28T15:52:04.408504+0000","last_clean":"2024-03-28T15:52:04.408504+0000","last_became_active":"2024-03-28T15:50:32.431295+0000","last_became_peered":"2024-03-28T15:50:32.431295+0000","last_unstale":"2024-03-28T15:52:04.408504+0000","last_undegraded":"2024-03-28T15:52:04.408504+0000","last_fullsized":"2024-03-28T15:52:04.408504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:46:12.055577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.350120+0000","last_change":"2024-03-28T15:50:36.459198+0000","last_active":"2024-03-28T15:51:35.350120+0000","last_peered":"2024-03-28T15:51:35.350120+0000","last_clean":"2024-03-28T15:51:35.350120+0000","last_became_active":"2024-03-28T15:50:36.458815+0000","last_became_peered":"2024-03-28T15:50:36.458815+0000","last_unstale":"2024-03-28T15:51:35.350120+0000","last_undegraded":"2024-03-28T15:51:35.350120+0000","last_fullsized":"2024-03-28T15:51:35.350120+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:19.302434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349163+0000","last_change":"2024-03-28T15:50:31.421319+0000","last_active":"2024-03-28T15:51:35.349163+0000","last_peered":"2024-03-28T15:51:35.349163+0000","last_clean":"2024-03-28T15:51:35.349163+0000","last_became_active":"2024-03-28T15:50:31.420399+0000","last_became_peered":"2024-03-28T15:50:31.420399+0000","last_unstale":"2024-03-28T15:51:35.349163+0000","last_undegraded":"2024-03-28T15:51:35.349163+0000","last_fullsized":"2024-03-28T15:51:35.349163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:45.594801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.414233+0000","last_change":"2024-03-28T15:50:32.436155+0000","last_active":"2024-03-28T15:52:04.414233+0000","last_peered":"2024-03-28T15:52:04.414233+0000","last_clean":"2024-03-28T15:52:04.414233+0000","last_became_active":"2024-03-28T15:50:32.435453+0000","last_became_peered":"2024-03-28T15:50:32.435453+0000","last_unstale":"2024-03-28T15:52:04.414233+0000","last_undegraded":"2024-03-28T15:52:04.414233+0000","last_fullsized":"2024-03-28T15:52:04.414233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:44.159744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.718568+0000","last_change":"2024-03-28T15:50:36.456103+0000","last_active":"2024-03-28T15:51:35.718568+0000","last_peered":"2024-03-28T15:51:35.718568+0000","last_clean":"2024-03-28T15:51:35.718568+0000","last_became_active":"2024-03-28T15:50:36.455724+0000","last_became_peered":"2024-03-28T15:50:36.455724+0000","last_unstale":"2024-03-28T15:51:35.718568+0000","last_undegraded":"2024-03-28T15:51:35.718568+0000","last_fullsized":"2024-03-28T15:51:35.718568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:30.874245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:06.291944+0000","last_change":"2024-03-28T15:50:31.423648+0000","last_active":"2024-03-28T15:52:06.291944+0000","last_peered":"2024-03-28T15:52:06.291944+0000","last_clean":"2024-03-28T15:52:06.291944+0000","last_became_active":"2024-03-28T15:50:31.423343+0000","last_became_peered":"2024-03-28T15:50:31.423343+0000","last_unstale":"2024-03-28T15:52:06.291944+0000","last_undegraded":"2024-03-28T15:52:06.291944+0000","last_fullsized":"2024-03-28T15:52:06.291944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:22.475804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417440+0000","last_change":"2024-03-28T15:50:32.432943+0000","last_active":"2024-03-28T15:52:04.417440+0000","last_peered":"2024-03-28T15:52:04.417440+0000","last_clean":"2024-03-28T15:52:04.417440+0000","last_became_active":"2024-03-28T15:50:32.432535+0000","last_became_peered":"2024-03-28T15:50:32.432535+0000","last_unstale":"2024-03-28T15:52:04.417440+0000","last_undegraded":"2024-03-28T15:52:04.417440+0000","last_fullsized":"2024-03-28T15:52:04.417440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:20:17.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.719765+0000","last_change":"2024-03-28T15:51:35.719765+0000","last_active":"2024-03-28T15:51:35.719765+0000","last_peered":"2024-03-28T15:51:35.719765+0000","last_clean":"2024-03-28T15:51:35.719765+0000","last_became_active":"2024-03-28T15:51:35.719380+0000","last_became_peered":"2024-03-28T15:51:35.719380+0000","last_unstale":"2024-03-28T15:51:35.719765+0000","last_undegraded":"2024-03-28T15:51:35.719765+0000","last_fullsized":"2024-03-28T15:51:35.719765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:28:56.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352914+0000","last_change":"2024-03-28T15:50:31.419043+0000","last_active":"2024-03-28T15:51:35.352914+0000","last_peered":"2024-03-28T15:51:35.352914+0000","last_clean":"2024-03-28T15:51:35.352914+0000","last_became_active":"2024-03-28T15:50:31.418722+0000","last_became_peered":"2024-03-28T15:50:31.418722+0000","last_unstale":"2024-03-28T15:51:35.352914+0000","last_undegraded":"2024-03-28T15:51:35.352914+0000","last_fullsized":"2024-03-28T15:51:35.352914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:10.512442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406546+0000","last_change":"2024-03-28T15:51:35.356071+0000","last_active":"2024-03-28T15:52:04.406546+0000","last_peered":"2024-03-28T15:52:04.406546+0000","last_clean":"2024-03-28T15:52:04.406546+0000","last_became_active":"2024-03-28T15:51:35.355695+0000","last_became_peered":"2024-03-28T15:51:35.355695+0000","last_unstale":"2024-03-28T15:52:04.406546+0000","last_undegraded":"2024-03-28T15:52:04.406546+0000","last_fullsized":"2024-03-28T15:52:04.406546+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:10.535552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351755+0000","last_change":"2024-03-28T15:50:36.455924+0000","last_active":"2024-03-28T15:51:35.351755+0000","last_peered":"2024-03-28T15:51:35.351755+0000","last_clean":"2024-03-28T15:51:35.351755+0000","last_became_active":"2024-03-28T15:50:36.455511+0000","last_became_peered":"2024-03-28T15:50:36.455511+0000","last_unstale":"2024-03-28T15:51:35.351755+0000","last_undegraded":"2024-03-28T15:51:35.351755+0000","last_fullsized":"2024-03-28T15:51:35.351755+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:48.039313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934572+0000","last_change":"2024-03-28T15:50:31.418393+0000","last_active":"2024-03-28T15:51:35.934572+0000","last_peered":"2024-03-28T15:51:35.934572+0000","last_clean":"2024-03-28T15:51:35.934572+0000","last_became_active":"2024-03-28T15:50:31.418086+0000","last_became_peered":"2024-03-28T15:50:31.418086+0000","last_unstale":"2024-03-28T15:51:35.934572+0000","last_undegraded":"2024-03-28T15:51:35.934572+0000","last_fullsized":"2024-03-28T15:51:35.934572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:44.882002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.411962+0000","last_change":"2024-03-28T15:50:32.429008+0000","last_active":"2024-03-28T15:52:04.411962+0000","last_peered":"2024-03-28T15:52:04.411962+0000","last_clean":"2024-03-28T15:52:04.411962+0000","last_became_active":"2024-03-28T15:50:32.428398+0000","last_became_peered":"2024-03-28T15:50:32.428398+0000","last_unstale":"2024-03-28T15:52:04.411962+0000","last_undegraded":"2024-03-28T15:52:04.411962+0000","last_fullsized":"2024-03-28T15:52:04.411962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:50.908576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.404183+0000","last_change":"2024-03-28T15:50:32.424785+0000","last_active":"2024-03-28T15:52:04.404183+0000","last_peered":"2024-03-28T15:52:04.404183+0000","last_clean":"2024-03-28T15:52:04.404183+0000","last_became_active":"2024-03-28T15:50:32.424134+0000","last_became_peered":"2024-03-28T15:50:32.424134+0000","last_unstale":"2024-03-28T15:52:04.404183+0000","last_undegraded":"2024-03-28T15:52:04.404183+0000","last_fullsized":"2024-03-28T15:52:04.404183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:20.842423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410620+0000","last_change":"2024-03-28T15:51:35.356185+0000","last_active":"2024-03-28T15:52:04.410620+0000","last_peered":"2024-03-28T15:52:04.410620+0000","last_clean":"2024-03-28T15:52:04.410620+0000","last_became_active":"2024-03-28T15:51:35.355709+0000","last_became_peered":"2024-03-28T15:51:35.355709+0000","last_unstale":"2024-03-28T15:52:04.410620+0000","last_undegraded":"2024-03-28T15:52:04.410620+0000","last_fullsized":"2024-03-28T15:52:04.410620+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:02:33.178070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.393753+0000","last_change":"2024-03-28T15:50:32.425448+0000","last_active":"2024-03-28T15:52:04.393753+0000","last_peered":"2024-03-28T15:52:04.393753+0000","last_clean":"2024-03-28T15:52:04.393753+0000","last_became_active":"2024-03-28T15:50:32.425148+0000","last_became_peered":"2024-03-28T15:50:32.425148+0000","last_unstale":"2024-03-28T15:52:04.393753+0000","last_undegraded":"2024-03-28T15:52:04.393753+0000","last_fullsized":"2024-03-28T15:52:04.393753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:31.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.411001+0000","last_change":"2024-03-28T15:50:32.433610+0000","last_active":"2024-03-28T15:52:04.411001+0000","last_peered":"2024-03-28T15:52:04.411001+0000","last_clean":"2024-03-28T15:52:04.411001+0000","last_became_active":"2024-03-28T15:50:32.433096+0000","last_became_peered":"2024-03-28T15:50:32.433096+0000","last_unstale":"2024-03-28T15:52:04.411001+0000","last_undegraded":"2024-03-28T15:52:04.411001+0000","last_fullsized":"2024-03-28T15:52:04.411001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:20.479797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402852+0000","last_change":"2024-03-28T15:50:32.428195+0000","last_active":"2024-03-28T15:52:04.402852+0000","last_peered":"2024-03-28T15:52:04.402852+0000","last_clean":"2024-03-28T15:52:04.402852+0000","last_became_active":"2024-03-28T15:50:32.427877+0000","last_became_peered":"2024-03-28T15:50:32.427877+0000","last_unstale":"2024-03-28T15:52:04.402852+0000","last_undegraded":"2024-03-28T15:52:04.402852+0000","last_fullsized":"2024-03-28T15:52:04.402852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:56.998688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406562+0000","last_change":"2024-03-28T15:50:32.420177+0000","last_active":"2024-03-28T15:52:04.406562+0000","last_peered":"2024-03-28T15:52:04.406562+0000","last_clean":"2024-03-28T15:52:04.406562+0000","last_became_active":"2024-03-28T15:50:32.419805+0000","last_became_peered":"2024-03-28T15:50:32.419805+0000","last_unstale":"2024-03-28T15:52:04.406562+0000","last_undegraded":"2024-03-28T15:52:04.406562+0000","last_fullsized":"2024-03-28T15:52:04.406562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:04:31.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403874+0000","last_change":"2024-03-28T15:50:32.417500+0000","last_active":"2024-03-28T15:52:04.403874+0000","last_peered":"2024-03-28T15:52:04.403874+0000","last_clean":"2024-03-28T15:52:04.403874+0000","last_became_active":"2024-03-28T15:50:32.416843+0000","last_became_peered":"2024-03-28T15:50:32.416843+0000","last_unstale":"2024-03-28T15:52:04.403874+0000","last_undegraded":"2024-03-28T15:52:04.403874+0000","last_fullsized":"2024-03-28T15:52:04.403874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:18:03.656816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403845+0000","last_change":"2024-03-28T15:50:32.428655+0000","last_active":"2024-03-28T15:52:04.403845+0000","last_peered":"2024-03-28T15:52:04.403845+0000","last_clean":"2024-03-28T15:52:04.403845+0000","last_became_active":"2024-03-28T15:50:32.428297+0000","last_became_peered":"2024-03-28T15:50:32.428297+0000","last_unstale":"2024-03-28T15:52:04.403845+0000","last_undegraded":"2024-03-28T15:52:04.403845+0000","last_fullsized":"2024-03-28T15:52:04.403845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:55:32.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.405148+0000","last_change":"2024-03-28T15:50:32.422012+0000","last_active":"2024-03-28T15:52:04.405148+0000","last_peered":"2024-03-28T15:52:04.405148+0000","last_clean":"2024-03-28T15:52:04.405148+0000","last_became_active":"2024-03-28T15:50:32.421465+0000","last_became_peered":"2024-03-28T15:50:32.421465+0000","last_unstale":"2024-03-28T15:52:04.405148+0000","last_undegraded":"2024-03-28T15:52:04.405148+0000","last_fullsized":"2024-03-28T15:52:04.405148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:35.235917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407293+0000","last_change":"2024-03-28T15:50:32.426624+0000","last_active":"2024-03-28T15:52:04.407293+0000","last_peered":"2024-03-28T15:52:04.407293+0000","last_clean":"2024-03-28T15:52:04.407293+0000","last_became_active":"2024-03-28T15:50:32.425307+0000","last_became_peered":"2024-03-28T15:50:32.425307+0000","last_unstale":"2024-03-28T15:52:04.407293+0000","last_undegraded":"2024-03-28T15:52:04.407293+0000","last_fullsized":"2024-03-28T15:52:04.407293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:05.332057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.420108+0000","last_change":"2024-03-28T15:50:32.430274+0000","last_active":"2024-03-28T15:52:04.420108+0000","last_peered":"2024-03-28T15:52:04.420108+0000","last_clean":"2024-03-28T15:52:04.420108+0000","last_became_active":"2024-03-28T15:50:32.429147+0000","last_became_peered":"2024-03-28T15:50:32.429147+0000","last_unstale":"2024-03-28T15:52:04.420108+0000","last_undegraded":"2024-03-28T15:52:04.420108+0000","last_fullsized":"2024-03-28T15:52:04.420108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:19.135249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.399624+0000","last_change":"2024-03-28T15:50:32.434933+0000","last_active":"2024-03-28T15:52:04.399624+0000","last_peered":"2024-03-28T15:52:04.399624+0000","last_clean":"2024-03-28T15:52:04.399624+0000","last_became_active":"2024-03-28T15:50:32.434612+0000","last_became_peered":"2024-03-28T15:50:32.434612+0000","last_unstale":"2024-03-28T15:52:04.399624+0000","last_undegraded":"2024-03-28T15:52:04.399624+0000","last_fullsized":"2024-03-28T15:52:04.399624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:41.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406746+0000","last_change":"2024-03-28T15:50:32.426181+0000","last_active":"2024-03-28T15:52:04.406746+0000","last_peered":"2024-03-28T15:52:04.406746+0000","last_clean":"2024-03-28T15:52:04.406746+0000","last_became_active":"2024-03-28T15:50:32.425866+0000","last_became_peered":"2024-03-28T15:50:32.425866+0000","last_unstale":"2024-03-28T15:52:04.406746+0000","last_undegraded":"2024-03-28T15:52:04.406746+0000","last_fullsized":"2024-03-28T15:52:04.406746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:42.519070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250168,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66074,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":734,"num_read_kb":444,"num_write":246,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":504484,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1154516,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":504414,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":504792,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299445,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":1101430,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":499918,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":560068,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":501374,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":560820,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":499626,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888132,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":503992,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":1097012,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:52:12.820 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-28T15:52:13.067 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:13.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:12 smithi023 ceph-mon[32448]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:13.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:12 smithi117 ceph-mon[36002]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:13.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:12 smithi120 ceph-mon[36826]: from='client.24959 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:14.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:13 smithi117 ceph-mon[36002]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T15:52:14.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:13 smithi120 ceph-mon[36826]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T15:52:14.406 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:13 smithi023 ceph-mon[32448]: pgmap v341: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 -- 172.21.15.23:0/2056472715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d810ec40 msgr2=0x7ff1d8111030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/2056472715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d810ec40 0x7ff1d8111030 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000b0a0 tx=0x7ff1d000b3b0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 -- 172.21.15.23:0/2056472715 shutdown_connections 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/2056472715 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d810ec40 0x7ff1d8111030 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/2056472715 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d8069e40 0x7ff1d806a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/2056472715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d806a760 0x7ff1d810e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 -- 172.21.15.23:0/2056472715 >> 172.21.15.23:0/2056472715 conn(0x7ff1d8077fc0 msgr2=0x7ff1d80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 -- 172.21.15.23:0/2056472715 shutdown_connections 2024-03-28T15:52:15.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.038+0000 7ff1df7d3700 1 -- 172.21.15.23:0/2056472715 wait complete. 2024-03-28T15:52:15.042 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 Processor -- start 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 -- start start 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d806a760 0x7ff1d81a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 0x7ff1d819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d8114650 con 0x7ff1d8069e40 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81144d0 con 0x7ff1d810ec40 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1df7d3700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d81147d0 con 0x7ff1d806a760 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1dd56f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1dcd6e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d806a760 0x7ff1d81a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.039+0000 7ff1dd56f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51934/0 (socket says 172.21.15.23:51934) 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 -- 172.21.15.23:0/292173264 learned_addr learned my addr 172.21.15.23:0/292173264 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:15.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 -- 172.21.15.23:0/292173264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d806a760 msgr2=0x7ff1d81a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d806a760 0x7ff1d81a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 -- 172.21.15.23:0/292173264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 msgr2=0x7ff1d819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1ddd70700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 0x7ff1d819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 0x7ff1d819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 -- 172.21.15.23:0/292173264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d819b0b0 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1ddd70700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 0x7ff1d819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1dd56f700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400e9c0 tx=0x7ff1d400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d400cbd0 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1d819b3a0 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d81a7660 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.040+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d400cd70 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.041+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4010430 con 0x7ff1d8069e40 2024-03-28T15:52:15.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.042+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1bc005330 con 0x7ff1d8069e40 2024-03-28T15:52:15.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.043+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff1d4004750 con 0x7ff1d8069e40 2024-03-28T15:52:15.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.043+0000 7ff1ce7fc700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 0x7ff1c40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:15.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.044+0000 7ff1dcd6e700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 0x7ff1c40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:15.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.044+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff1d40cef40 con 0x7ff1d8069e40 2024-03-28T15:52:15.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.044+0000 7ff1dcd6e700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 0x7ff1c40a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81025f0 tx=0x7ff1c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:15.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.048+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff1d409bc80 con 0x7ff1d8069e40 2024-03-28T15:52:15.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.320+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1bc000bc0 con 0x7ff1c40a2910 2024-03-28T15:52:15.335 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.334+0000 7ff1ce7fc700 1 -- 172.21.15.23:0/292173264 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340777 (secure 0 0 0) 0x7ff1bc000bc0 con 0x7ff1c40a2910 2024-03-28T15:52:15.341 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:15.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 msgr2=0x7ff1c40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:15.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 0x7ff1c40a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff1d81025f0 tx=0x7ff1c8009260 comp rx=0 tx=0).stop 2024-03-28T15:52:15.343 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 msgr2=0x7ff1d81a0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400e9c0 tx=0x7ff1d400ecd0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 shutdown_connections 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7ff1c40a2910 0x7ff1c40a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d810ec40 0x7ff1d819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff1d8069e40 0x7ff1d81a0790 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 --2- 172.21.15.23:0/292173264 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff1d806a760 0x7ff1d81a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 >> 172.21.15.23:0/292173264 conn(0x7ff1d8077fc0 msgr2=0x7ff1d81100f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 shutdown_connections 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:15.338+0000 7ff1df7d3700 1 -- 172.21.15.23:0/292173264 wait complete. 2024-03-28T15:52:15.344 INFO:teuthology.orchestra.run.smithi023.stderr:dumped all 2024-03-28T15:52:16.024 INFO:teuthology.orchestra.run.smithi023.stdout:{"pg_ready":true,"pg_map":{"version":342,"stamp":"2024-03-28T15:52:14.393679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656676,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":900,"num_read_kb":573,"num_write":429,"num_write_kb":3080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369864,"kb_used_data":12392,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571960,"statfs":{"total":1151940427776,"available":1151561687040,"internally_reserved":0,"allocated":12689408,"data_stored":7992446,"data_compressed":357081,"data_compressed_allocated":479232,"data_compressed_original":2457600,"omap_allocated":19140,"internal_metadata":365933884},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":121,"num_read_kb":61,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"3.3f","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419695+0000","last_change":"2024-03-28T15:50:32.417721+0000","last_active":"2024-03-28T15:52:04.419695+0000","last_peered":"2024-03-28T15:52:04.419695+0000","last_clean":"2024-03-28T15:52:04.419695+0000","last_became_active":"2024-03-28T15:50:32.417397+0000","last_became_peered":"2024-03-28T15:50:32.417397+0000","last_unstale":"2024-03-28T15:52:04.419695+0000","last_undegraded":"2024-03-28T15:52:04.419695+0000","last_fullsized":"2024-03-28T15:52:04.419695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:49:18.755144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409197+0000","last_change":"2024-03-28T15:50:32.419861+0000","last_active":"2024-03-28T15:52:04.409197+0000","last_peered":"2024-03-28T15:52:04.409197+0000","last_clean":"2024-03-28T15:52:04.409197+0000","last_became_active":"2024-03-28T15:50:32.419472+0000","last_became_peered":"2024-03-28T15:50:32.419472+0000","last_unstale":"2024-03-28T15:52:04.409197+0000","last_undegraded":"2024-03-28T15:52:04.409197+0000","last_fullsized":"2024-03-28T15:52:04.409197+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:20.452492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417286+0000","last_change":"2024-03-28T15:50:32.425423+0000","last_active":"2024-03-28T15:52:04.417286+0000","last_peered":"2024-03-28T15:52:04.417286+0000","last_clean":"2024-03-28T15:52:04.417286+0000","last_became_active":"2024-03-28T15:50:32.423503+0000","last_became_peered":"2024-03-28T15:50:32.423503+0000","last_unstale":"2024-03-28T15:52:04.417286+0000","last_undegraded":"2024-03-28T15:52:04.417286+0000","last_fullsized":"2024-03-28T15:52:04.417286+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:32:41.518085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408328+0000","last_change":"2024-03-28T15:50:32.426795+0000","last_active":"2024-03-28T15:52:04.408328+0000","last_peered":"2024-03-28T15:52:04.408328+0000","last_clean":"2024-03-28T15:52:04.408328+0000","last_became_active":"2024-03-28T15:50:32.426495+0000","last_became_peered":"2024-03-28T15:50:32.426495+0000","last_unstale":"2024-03-28T15:52:04.408328+0000","last_undegraded":"2024-03-28T15:52:04.408328+0000","last_fullsized":"2024-03-28T15:52:04.408328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:57:55.850464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.388993+0000","last_change":"2024-03-28T15:50:32.432258+0000","last_active":"2024-03-28T15:52:04.388993+0000","last_peered":"2024-03-28T15:52:04.388993+0000","last_clean":"2024-03-28T15:52:04.388993+0000","last_became_active":"2024-03-28T15:50:32.431840+0000","last_became_peered":"2024-03-28T15:50:32.431840+0000","last_unstale":"2024-03-28T15:52:04.388993+0000","last_undegraded":"2024-03-28T15:52:04.388993+0000","last_fullsized":"2024-03-28T15:52:04.388993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:41:30.878251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410749+0000","last_change":"2024-03-28T15:50:32.421362+0000","last_active":"2024-03-28T15:52:04.410749+0000","last_peered":"2024-03-28T15:52:04.410749+0000","last_clean":"2024-03-28T15:52:04.410749+0000","last_became_active":"2024-03-28T15:50:32.421039+0000","last_became_peered":"2024-03-28T15:50:32.421039+0000","last_unstale":"2024-03-28T15:52:04.410749+0000","last_undegraded":"2024-03-28T15:52:04.410749+0000","last_fullsized":"2024-03-28T15:52:04.410749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:07:01.590154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406923+0000","last_change":"2024-03-28T15:50:32.423923+0000","last_active":"2024-03-28T15:52:04.406923+0000","last_peered":"2024-03-28T15:52:04.406923+0000","last_clean":"2024-03-28T15:52:04.406923+0000","last_became_active":"2024-03-28T15:50:32.423147+0000","last_became_peered":"2024-03-28T15:50:32.423147+0000","last_unstale":"2024-03-28T15:52:04.406923+0000","last_undegraded":"2024-03-28T15:52:04.406923+0000","last_fullsized":"2024-03-28T15:52:04.406923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:42:18.526091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"92'9","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410627+0000","last_change":"2024-03-28T15:50:32.428936+0000","last_active":"2024-03-28T15:52:04.410627+0000","last_peered":"2024-03-28T15:52:04.410627+0000","last_clean":"2024-03-28T15:52:04.410627+0000","last_became_active":"2024-03-28T15:50:32.427438+0000","last_became_peered":"2024-03-28T15:50:32.427438+0000","last_unstale":"2024-03-28T15:52:04.410627+0000","last_undegraded":"2024-03-28T15:52:04.410627+0000","last_fullsized":"2024-03-28T15:52:04.410627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:29:20.788775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419692+0000","last_change":"2024-03-28T15:50:32.426691+0000","last_active":"2024-03-28T15:52:04.419692+0000","last_peered":"2024-03-28T15:52:04.419692+0000","last_clean":"2024-03-28T15:52:04.419692+0000","last_became_active":"2024-03-28T15:50:32.426408+0000","last_became_peered":"2024-03-28T15:50:32.426408+0000","last_unstale":"2024-03-28T15:52:04.419692+0000","last_undegraded":"2024-03-28T15:52:04.419692+0000","last_fullsized":"2024-03-28T15:52:04.419692+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:09:45.911656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"92'16","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409200+0000","last_change":"2024-03-28T15:50:32.428322+0000","last_active":"2024-03-28T15:52:04.409200+0000","last_peered":"2024-03-28T15:52:04.409200+0000","last_clean":"2024-03-28T15:52:04.409200+0000","last_became_active":"2024-03-28T15:50:32.428001+0000","last_became_peered":"2024-03-28T15:50:32.428001+0000","last_unstale":"2024-03-28T15:52:04.409200+0000","last_undegraded":"2024-03-28T15:52:04.409200+0000","last_fullsized":"2024-03-28T15:52:04.409200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:18:00.795303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417305+0000","last_change":"2024-03-28T15:50:32.435072+0000","last_active":"2024-03-28T15:52:04.417305+0000","last_peered":"2024-03-28T15:52:04.417305+0000","last_clean":"2024-03-28T15:52:04.417305+0000","last_became_active":"2024-03-28T15:50:32.434757+0000","last_became_peered":"2024-03-28T15:50:32.434757+0000","last_unstale":"2024-03-28T15:52:04.417305+0000","last_undegraded":"2024-03-28T15:52:04.417305+0000","last_fullsized":"2024-03-28T15:52:04.417305+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:45:34.357151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.405151+0000","last_change":"2024-03-28T15:50:32.428312+0000","last_active":"2024-03-28T15:52:04.405151+0000","last_peered":"2024-03-28T15:52:04.405151+0000","last_clean":"2024-03-28T15:52:04.405151+0000","last_became_active":"2024-03-28T15:50:32.428020+0000","last_became_peered":"2024-03-28T15:50:32.428020+0000","last_unstale":"2024-03-28T15:52:04.405151+0000","last_undegraded":"2024-03-28T15:52:04.405151+0000","last_fullsized":"2024-03-28T15:52:04.405151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:11.841606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398999+0000","last_change":"2024-03-28T15:50:32.422259+0000","last_active":"2024-03-28T15:52:04.398999+0000","last_peered":"2024-03-28T15:52:04.398999+0000","last_clean":"2024-03-28T15:52:04.398999+0000","last_became_active":"2024-03-28T15:50:32.421181+0000","last_became_peered":"2024-03-28T15:50:32.421181+0000","last_unstale":"2024-03-28T15:52:04.398999+0000","last_undegraded":"2024-03-28T15:52:04.398999+0000","last_fullsized":"2024-03-28T15:52:04.398999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:16:40.403642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406213+0000","last_change":"2024-03-28T15:50:32.430402+0000","last_active":"2024-03-28T15:52:04.406213+0000","last_peered":"2024-03-28T15:52:04.406213+0000","last_clean":"2024-03-28T15:52:04.406213+0000","last_became_active":"2024-03-28T15:50:32.429978+0000","last_became_peered":"2024-03-28T15:50:32.429978+0000","last_unstale":"2024-03-28T15:52:04.406213+0000","last_undegraded":"2024-03-28T15:52:04.406213+0000","last_fullsized":"2024-03-28T15:52:04.406213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:37:06.407373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"92'7","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410258+0000","last_change":"2024-03-28T15:51:35.358518+0000","last_active":"2024-03-28T15:52:04.410258+0000","last_peered":"2024-03-28T15:52:04.410258+0000","last_clean":"2024-03-28T15:52:04.410258+0000","last_became_active":"2024-03-28T15:51:35.358203+0000","last_became_peered":"2024-03-28T15:51:35.358203+0000","last_unstale":"2024-03-28T15:52:04.410258+0000","last_undegraded":"2024-03-28T15:52:04.410258+0000","last_fullsized":"2024-03-28T15:52:04.410258+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:20:04.984476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419377+0000","last_change":"2024-03-28T15:50:32.429542+0000","last_active":"2024-03-28T15:52:04.419377+0000","last_peered":"2024-03-28T15:52:04.419377+0000","last_clean":"2024-03-28T15:52:04.419377+0000","last_became_active":"2024-03-28T15:50:32.427264+0000","last_became_peered":"2024-03-28T15:50:32.427264+0000","last_unstale":"2024-03-28T15:52:04.419377+0000","last_undegraded":"2024-03-28T15:52:04.419377+0000","last_fullsized":"2024-03-28T15:52:04.419377+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:56:54.272678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.420043+0000","last_change":"2024-03-28T15:50:32.418775+0000","last_active":"2024-03-28T15:52:04.420043+0000","last_peered":"2024-03-28T15:52:04.420043+0000","last_clean":"2024-03-28T15:52:04.420043+0000","last_became_active":"2024-03-28T15:50:32.418453+0000","last_became_peered":"2024-03-28T15:50:32.418453+0000","last_unstale":"2024-03-28T15:52:04.420043+0000","last_undegraded":"2024-03-28T15:52:04.420043+0000","last_fullsized":"2024-03-28T15:52:04.420043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:54:26.061231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402093+0000","last_change":"2024-03-28T15:50:32.429111+0000","last_active":"2024-03-28T15:52:04.402093+0000","last_peered":"2024-03-28T15:52:04.402093+0000","last_clean":"2024-03-28T15:52:04.402093+0000","last_became_active":"2024-03-28T15:50:32.428036+0000","last_became_peered":"2024-03-28T15:50:32.428036+0000","last_unstale":"2024-03-28T15:52:04.402093+0000","last_undegraded":"2024-03-28T15:52:04.402093+0000","last_fullsized":"2024-03-28T15:52:04.402093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:09.272224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.393050+0000","last_change":"2024-03-28T15:50:32.422982+0000","last_active":"2024-03-28T15:52:04.393050+0000","last_peered":"2024-03-28T15:52:04.393050+0000","last_clean":"2024-03-28T15:52:04.393050+0000","last_became_active":"2024-03-28T15:50:32.422667+0000","last_became_peered":"2024-03-28T15:50:32.422667+0000","last_unstale":"2024-03-28T15:52:04.393050+0000","last_undegraded":"2024-03-28T15:52:04.393050+0000","last_fullsized":"2024-03-28T15:52:04.393050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:12:51.445603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351583+0000","last_change":"2024-03-28T15:50:36.462626+0000","last_active":"2024-03-28T15:51:35.351583+0000","last_peered":"2024-03-28T15:51:35.351583+0000","last_clean":"2024-03-28T15:51:35.351583+0000","last_became_active":"2024-03-28T15:50:36.462168+0000","last_became_peered":"2024-03-28T15:50:36.462168+0000","last_unstale":"2024-03-28T15:51:35.351583+0000","last_undegraded":"2024-03-28T15:51:35.351583+0000","last_fullsized":"2024-03-28T15:51:35.351583+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:02:59.676350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"90'13","reported_seq":114,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:12.306808+0000","last_change":"2024-03-28T15:50:31.425778+0000","last_active":"2024-03-28T15:52:12.306808+0000","last_peered":"2024-03-28T15:52:12.306808+0000","last_clean":"2024-03-28T15:52:12.306808+0000","last_became_active":"2024-03-28T15:50:31.424946+0000","last_became_peered":"2024-03-28T15:50:31.424946+0000","last_unstale":"2024-03-28T15:52:12.306808+0000","last_undegraded":"2024-03-28T15:52:12.306808+0000","last_fullsized":"2024-03-28T15:52:12.306808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:50.444459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.401877+0000","last_change":"2024-03-28T15:50:32.431201+0000","last_active":"2024-03-28T15:52:04.401877+0000","last_peered":"2024-03-28T15:52:04.401877+0000","last_clean":"2024-03-28T15:52:04.401877+0000","last_became_active":"2024-03-28T15:50:32.428970+0000","last_became_peered":"2024-03-28T15:50:32.428970+0000","last_unstale":"2024-03-28T15:52:04.401877+0000","last_undegraded":"2024-03-28T15:52:04.401877+0000","last_fullsized":"2024-03-28T15:52:04.401877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:38:36.966552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349277+0000","last_change":"2024-03-28T15:50:36.457727+0000","last_active":"2024-03-28T15:51:35.349277+0000","last_peered":"2024-03-28T15:51:35.349277+0000","last_clean":"2024-03-28T15:51:35.349277+0000","last_became_active":"2024-03-28T15:50:36.457322+0000","last_became_peered":"2024-03-28T15:50:36.457322+0000","last_unstale":"2024-03-28T15:51:35.349277+0000","last_undegraded":"2024-03-28T15:51:35.349277+0000","last_fullsized":"2024-03-28T15:51:35.349277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:10:09.138564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"90'2","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.852963+0000","last_change":"2024-03-28T15:50:31.418890+0000","last_active":"2024-03-28T15:51:35.852963+0000","last_peered":"2024-03-28T15:51:35.852963+0000","last_clean":"2024-03-28T15:51:35.852963+0000","last_became_active":"2024-03-28T15:50:31.418445+0000","last_became_peered":"2024-03-28T15:50:31.418445+0000","last_unstale":"2024-03-28T15:51:35.852963+0000","last_undegraded":"2024-03-28T15:51:35.852963+0000","last_fullsized":"2024-03-28T15:51:35.852963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:01:15.157292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419869+0000","last_change":"2024-03-28T15:50:32.434362+0000","last_active":"2024-03-28T15:52:04.419869+0000","last_peered":"2024-03-28T15:52:04.419869+0000","last_clean":"2024-03-28T15:52:04.419869+0000","last_became_active":"2024-03-28T15:50:32.433577+0000","last_became_peered":"2024-03-28T15:50:32.433577+0000","last_unstale":"2024-03-28T15:52:04.419869+0000","last_undegraded":"2024-03-28T15:52:04.419869+0000","last_fullsized":"2024-03-28T15:52:04.419869+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:34:13.424149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349380+0000","last_change":"2024-03-28T15:50:36.460311+0000","last_active":"2024-03-28T15:51:35.349380+0000","last_peered":"2024-03-28T15:51:35.349380+0000","last_clean":"2024-03-28T15:51:35.349380+0000","last_became_active":"2024-03-28T15:50:36.459934+0000","last_became_peered":"2024-03-28T15:50:36.459934+0000","last_unstale":"2024-03-28T15:51:35.349380+0000","last_undegraded":"2024-03-28T15:51:35.349380+0000","last_fullsized":"2024-03-28T15:51:35.349380+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:12:11.299668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"90'12","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349737+0000","last_change":"2024-03-28T15:50:31.421609+0000","last_active":"2024-03-28T15:51:35.349737+0000","last_peered":"2024-03-28T15:51:35.349737+0000","last_clean":"2024-03-28T15:51:35.349737+0000","last_became_active":"2024-03-28T15:50:31.420979+0000","last_became_peered":"2024-03-28T15:50:31.420979+0000","last_unstale":"2024-03-28T15:51:35.349737+0000","last_undegraded":"2024-03-28T15:51:35.349737+0000","last_fullsized":"2024-03-28T15:51:35.349737+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:44:42.974359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419908+0000","last_change":"2024-03-28T15:50:32.432909+0000","last_active":"2024-03-28T15:52:04.419908+0000","last_peered":"2024-03-28T15:52:04.419908+0000","last_clean":"2024-03-28T15:52:04.419908+0000","last_became_active":"2024-03-28T15:50:32.432167+0000","last_became_peered":"2024-03-28T15:50:32.432167+0000","last_unstale":"2024-03-28T15:52:04.419908+0000","last_undegraded":"2024-03-28T15:52:04.419908+0000","last_fullsized":"2024-03-28T15:52:04.419908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:26:20.168299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349303+0000","last_change":"2024-03-28T15:50:36.459184+0000","last_active":"2024-03-28T15:51:35.349303+0000","last_peered":"2024-03-28T15:51:35.349303+0000","last_clean":"2024-03-28T15:51:35.349303+0000","last_became_active":"2024-03-28T15:50:36.458807+0000","last_became_peered":"2024-03-28T15:50:36.458807+0000","last_unstale":"2024-03-28T15:51:35.349303+0000","last_undegraded":"2024-03-28T15:51:35.349303+0000","last_fullsized":"2024-03-28T15:51:35.349303+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:45:51.954748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"90'5","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349302+0000","last_change":"2024-03-28T15:50:31.422344+0000","last_active":"2024-03-28T15:51:35.349302+0000","last_peered":"2024-03-28T15:51:35.349302+0000","last_clean":"2024-03-28T15:51:35.349302+0000","last_became_active":"2024-03-28T15:50:31.421418+0000","last_became_peered":"2024-03-28T15:50:31.421418+0000","last_unstale":"2024-03-28T15:51:35.349302+0000","last_undegraded":"2024-03-28T15:51:35.349302+0000","last_fullsized":"2024-03-28T15:51:35.349302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:17:11.341382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"92'15","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.414157+0000","last_change":"2024-03-28T15:50:32.428599+0000","last_active":"2024-03-28T15:52:04.414157+0000","last_peered":"2024-03-28T15:52:04.414157+0000","last_clean":"2024-03-28T15:52:04.414157+0000","last_became_active":"2024-03-28T15:50:32.427823+0000","last_became_peered":"2024-03-28T15:50:32.427823+0000","last_unstale":"2024-03-28T15:52:04.414157+0000","last_undegraded":"2024-03-28T15:52:04.414157+0000","last_fullsized":"2024-03-28T15:52:04.414157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:03:18.244229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351764+0000","last_change":"2024-03-28T15:50:36.451717+0000","last_active":"2024-03-28T15:51:35.351764+0000","last_peered":"2024-03-28T15:51:35.351764+0000","last_clean":"2024-03-28T15:51:35.351764+0000","last_became_active":"2024-03-28T15:50:36.451217+0000","last_became_peered":"2024-03-28T15:50:36.451217+0000","last_unstale":"2024-03-28T15:51:35.351764+0000","last_undegraded":"2024-03-28T15:51:35.351764+0000","last_fullsized":"2024-03-28T15:51:35.351764+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:29:05.556180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"92'7","reported_seq":109,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:10.296204+0000","last_change":"2024-03-28T15:50:31.419469+0000","last_active":"2024-03-28T15:52:10.296204+0000","last_peered":"2024-03-28T15:52:10.296204+0000","last_clean":"2024-03-28T15:52:10.296204+0000","last_became_active":"2024-03-28T15:50:31.419162+0000","last_became_peered":"2024-03-28T15:50:31.419162+0000","last_unstale":"2024-03-28T15:52:10.296204+0000","last_undegraded":"2024-03-28T15:52:10.296204+0000","last_fullsized":"2024-03-28T15:52:10.296204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:05:25.697846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398133+0000","last_change":"2024-03-28T15:50:32.422764+0000","last_active":"2024-03-28T15:52:04.398133+0000","last_peered":"2024-03-28T15:52:04.398133+0000","last_clean":"2024-03-28T15:52:04.398133+0000","last_became_active":"2024-03-28T15:50:32.422368+0000","last_became_peered":"2024-03-28T15:50:32.422368+0000","last_unstale":"2024-03-28T15:52:04.398133+0000","last_undegraded":"2024-03-28T15:52:04.398133+0000","last_fullsized":"2024-03-28T15:52:04.398133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:37:50.195055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352741+0000","last_change":"2024-03-28T15:50:36.453526+0000","last_active":"2024-03-28T15:51:35.352741+0000","last_peered":"2024-03-28T15:51:35.352741+0000","last_clean":"2024-03-28T15:51:35.352741+0000","last_became_active":"2024-03-28T15:50:36.452099+0000","last_became_peered":"2024-03-28T15:50:36.452099+0000","last_unstale":"2024-03-28T15:51:35.352741+0000","last_undegraded":"2024-03-28T15:51:35.352741+0000","last_fullsized":"2024-03-28T15:51:35.352741+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:55:06.379921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.564402+0000","last_change":"2024-03-28T15:50:31.423636+0000","last_active":"2024-03-28T15:51:35.564402+0000","last_peered":"2024-03-28T15:51:35.564402+0000","last_clean":"2024-03-28T15:51:35.564402+0000","last_became_active":"2024-03-28T15:50:31.423303+0000","last_became_peered":"2024-03-28T15:50:31.423303+0000","last_unstale":"2024-03-28T15:51:35.564402+0000","last_undegraded":"2024-03-28T15:51:35.564402+0000","last_fullsized":"2024-03-28T15:51:35.564402+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:05:57.666490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410457+0000","last_change":"2024-03-28T15:50:32.428292+0000","last_active":"2024-03-28T15:52:04.410457+0000","last_peered":"2024-03-28T15:52:04.410457+0000","last_clean":"2024-03-28T15:52:04.410457+0000","last_became_active":"2024-03-28T15:50:32.427950+0000","last_became_peered":"2024-03-28T15:50:32.427950+0000","last_unstale":"2024-03-28T15:52:04.410457+0000","last_undegraded":"2024-03-28T15:52:04.410457+0000","last_fullsized":"2024-03-28T15:52:04.410457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:09:43.040690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353828+0000","last_change":"2024-03-28T15:50:36.461197+0000","last_active":"2024-03-28T15:51:35.353828+0000","last_peered":"2024-03-28T15:51:35.353828+0000","last_clean":"2024-03-28T15:51:35.353828+0000","last_became_active":"2024-03-28T15:50:36.460763+0000","last_became_peered":"2024-03-28T15:50:36.460763+0000","last_unstale":"2024-03-28T15:51:35.353828+0000","last_undegraded":"2024-03-28T15:51:35.353828+0000","last_fullsized":"2024-03-28T15:51:35.353828+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:06:23.500309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"90'3","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934556+0000","last_change":"2024-03-28T15:50:31.417000+0000","last_active":"2024-03-28T15:51:35.934556+0000","last_peered":"2024-03-28T15:51:35.934556+0000","last_clean":"2024-03-28T15:51:35.934556+0000","last_became_active":"2024-03-28T15:50:31.416652+0000","last_became_peered":"2024-03-28T15:50:31.416652+0000","last_unstale":"2024-03-28T15:51:35.934556+0000","last_undegraded":"2024-03-28T15:51:35.934556+0000","last_fullsized":"2024-03-28T15:51:35.934556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:06:55.338636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"92'8","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409999+0000","last_change":"2024-03-28T15:51:35.936192+0000","last_active":"2024-03-28T15:52:04.409999+0000","last_peered":"2024-03-28T15:52:04.409999+0000","last_clean":"2024-03-28T15:52:04.409999+0000","last_became_active":"2024-03-28T15:51:35.935863+0000","last_became_peered":"2024-03-28T15:51:35.935863+0000","last_unstale":"2024-03-28T15:52:04.409999+0000","last_undegraded":"2024-03-28T15:52:04.409999+0000","last_fullsized":"2024-03-28T15:52:04.409999+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:30:38.502087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353387+0000","last_change":"2024-03-28T15:50:36.460876+0000","last_active":"2024-03-28T15:51:35.353387+0000","last_peered":"2024-03-28T15:51:35.353387+0000","last_clean":"2024-03-28T15:51:35.353387+0000","last_became_active":"2024-03-28T15:50:36.460452+0000","last_became_peered":"2024-03-28T15:50:36.460452+0000","last_unstale":"2024-03-28T15:51:35.353387+0000","last_undegraded":"2024-03-28T15:51:35.353387+0000","last_fullsized":"2024-03-28T15:51:35.353387+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:08:36.260861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351897+0000","last_change":"2024-03-28T15:50:31.417925+0000","last_active":"2024-03-28T15:51:35.351897+0000","last_peered":"2024-03-28T15:51:35.351897+0000","last_clean":"2024-03-28T15:51:35.351897+0000","last_became_active":"2024-03-28T15:50:31.417615+0000","last_became_peered":"2024-03-28T15:50:31.417615+0000","last_unstale":"2024-03-28T15:51:35.351897+0000","last_undegraded":"2024-03-28T15:51:35.351897+0000","last_fullsized":"2024-03-28T15:51:35.351897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:33:31.208143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408474+0000","last_change":"2024-03-28T15:50:32.423511+0000","last_active":"2024-03-28T15:52:04.408474+0000","last_peered":"2024-03-28T15:52:04.408474+0000","last_clean":"2024-03-28T15:52:04.408474+0000","last_became_active":"2024-03-28T15:50:32.423194+0000","last_became_peered":"2024-03-28T15:50:32.423194+0000","last_unstale":"2024-03-28T15:52:04.408474+0000","last_undegraded":"2024-03-28T15:52:04.408474+0000","last_fullsized":"2024-03-28T15:52:04.408474+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:46:44.170488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.853350+0000","last_change":"2024-03-28T15:50:36.465051+0000","last_active":"2024-03-28T15:51:35.853350+0000","last_peered":"2024-03-28T15:51:35.853350+0000","last_clean":"2024-03-28T15:51:35.853350+0000","last_became_active":"2024-03-28T15:50:36.464579+0000","last_became_peered":"2024-03-28T15:50:36.464579+0000","last_unstale":"2024-03-28T15:51:35.853350+0000","last_undegraded":"2024-03-28T15:51:35.853350+0000","last_fullsized":"2024-03-28T15:51:35.853350+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:40:56.769485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"92'5","reported_seq":174,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:09.294631+0000","last_change":"2024-03-28T15:50:31.429484+0000","last_active":"2024-03-28T15:52:09.294631+0000","last_peered":"2024-03-28T15:52:09.294631+0000","last_clean":"2024-03-28T15:52:09.294631+0000","last_became_active":"2024-03-28T15:50:31.429206+0000","last_became_peered":"2024-03-28T15:50:31.429206+0000","last_unstale":"2024-03-28T15:52:09.294631+0000","last_undegraded":"2024-03-28T15:52:09.294631+0000","last_fullsized":"2024-03-28T15:52:09.294631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:14:05.599043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.419336+0000","last_change":"2024-03-28T15:50:32.421813+0000","last_active":"2024-03-28T15:52:04.419336+0000","last_peered":"2024-03-28T15:52:04.419336+0000","last_clean":"2024-03-28T15:52:04.419336+0000","last_became_active":"2024-03-28T15:50:32.421510+0000","last_became_peered":"2024-03-28T15:50:32.421510+0000","last_unstale":"2024-03-28T15:52:04.419336+0000","last_undegraded":"2024-03-28T15:52:04.419336+0000","last_fullsized":"2024-03-28T15:52:04.419336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:50:06.314487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.348840+0000","last_change":"2024-03-28T15:50:36.464148+0000","last_active":"2024-03-28T15:51:35.348840+0000","last_peered":"2024-03-28T15:51:35.348840+0000","last_clean":"2024-03-28T15:51:35.348840+0000","last_became_active":"2024-03-28T15:50:36.463696+0000","last_became_peered":"2024-03-28T15:50:36.463696+0000","last_unstale":"2024-03-28T15:51:35.348840+0000","last_undegraded":"2024-03-28T15:51:35.348840+0000","last_fullsized":"2024-03-28T15:51:35.348840+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:57.733727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353228+0000","last_change":"2024-03-28T15:50:31.422969+0000","last_active":"2024-03-28T15:51:35.353228+0000","last_peered":"2024-03-28T15:51:35.353228+0000","last_clean":"2024-03-28T15:51:35.353228+0000","last_became_active":"2024-03-28T15:50:31.422664+0000","last_became_peered":"2024-03-28T15:50:31.422664+0000","last_unstale":"2024-03-28T15:51:35.353228+0000","last_undegraded":"2024-03-28T15:51:35.353228+0000","last_fullsized":"2024-03-28T15:51:35.353228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:04:24.469105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"92'9","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407536+0000","last_change":"2024-03-28T15:51:35.357854+0000","last_active":"2024-03-28T15:52:04.407536+0000","last_peered":"2024-03-28T15:52:04.407536+0000","last_clean":"2024-03-28T15:52:04.407536+0000","last_became_active":"2024-03-28T15:51:35.357471+0000","last_became_peered":"2024-03-28T15:51:35.357471+0000","last_unstale":"2024-03-28T15:52:04.407536+0000","last_undegraded":"2024-03-28T15:52:04.407536+0000","last_fullsized":"2024-03-28T15:52:04.407536+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:51:58.970860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349655+0000","last_change":"2024-03-28T15:50:36.457721+0000","last_active":"2024-03-28T15:51:35.349655+0000","last_peered":"2024-03-28T15:51:35.349655+0000","last_clean":"2024-03-28T15:51:35.349655+0000","last_became_active":"2024-03-28T15:50:36.457028+0000","last_became_peered":"2024-03-28T15:50:36.457028+0000","last_unstale":"2024-03-28T15:51:35.349655+0000","last_undegraded":"2024-03-28T15:51:35.349655+0000","last_fullsized":"2024-03-28T15:51:35.349655+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:41:47.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352729+0000","last_change":"2024-03-28T15:50:31.418265+0000","last_active":"2024-03-28T15:51:35.352729+0000","last_peered":"2024-03-28T15:51:35.352729+0000","last_clean":"2024-03-28T15:51:35.352729+0000","last_became_active":"2024-03-28T15:50:31.417944+0000","last_became_peered":"2024-03-28T15:50:31.417944+0000","last_unstale":"2024-03-28T15:51:35.352729+0000","last_undegraded":"2024-03-28T15:51:35.352729+0000","last_fullsized":"2024-03-28T15:51:35.352729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:47:58.633512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.421435+0000","last_change":"2024-03-28T15:50:32.433180+0000","last_active":"2024-03-28T15:52:04.421435+0000","last_peered":"2024-03-28T15:52:04.421435+0000","last_clean":"2024-03-28T15:52:04.421435+0000","last_became_active":"2024-03-28T15:50:32.432598+0000","last_became_peered":"2024-03-28T15:50:32.432598+0000","last_unstale":"2024-03-28T15:52:04.421435+0000","last_undegraded":"2024-03-28T15:52:04.421435+0000","last_fullsized":"2024-03-28T15:52:04.421435+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:27:16.051480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.348851+0000","last_change":"2024-03-28T15:50:36.453471+0000","last_active":"2024-03-28T15:51:35.348851+0000","last_peered":"2024-03-28T15:51:35.348851+0000","last_clean":"2024-03-28T15:51:35.348851+0000","last_became_active":"2024-03-28T15:50:36.453070+0000","last_became_peered":"2024-03-28T15:50:36.453070+0000","last_unstale":"2024-03-28T15:51:35.348851+0000","last_undegraded":"2024-03-28T15:51:35.348851+0000","last_fullsized":"2024-03-28T15:51:35.348851+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:36:55.417086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"92'36","reported_seq":213,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:09.296397+0000","last_change":"2024-03-28T15:50:31.419585+0000","last_active":"2024-03-28T15:52:09.296397+0000","last_peered":"2024-03-28T15:52:09.296397+0000","last_clean":"2024-03-28T15:52:09.296397+0000","last_became_active":"2024-03-28T15:50:31.419241+0000","last_became_peered":"2024-03-28T15:50:31.419241+0000","last_unstale":"2024-03-28T15:52:09.296397+0000","last_undegraded":"2024-03-28T15:52:09.296397+0000","last_fullsized":"2024-03-28T15:52:09.296397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:34:17.438115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59120,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":150,"num_read_kb":133,"num_write":38,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"92'6","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409196+0000","last_change":"2024-03-28T15:50:32.436526+0000","last_active":"2024-03-28T15:52:04.409196+0000","last_peered":"2024-03-28T15:52:04.409196+0000","last_clean":"2024-03-28T15:52:04.409196+0000","last_became_active":"2024-03-28T15:50:32.436211+0000","last_became_peered":"2024-03-28T15:50:32.436211+0000","last_unstale":"2024-03-28T15:52:04.409196+0000","last_undegraded":"2024-03-28T15:52:04.409196+0000","last_fullsized":"2024-03-28T15:52:04.409196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:48:04.424281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353122+0000","last_change":"2024-03-28T15:50:36.460308+0000","last_active":"2024-03-28T15:51:35.353122+0000","last_peered":"2024-03-28T15:51:35.353122+0000","last_clean":"2024-03-28T15:51:35.353122+0000","last_became_active":"2024-03-28T15:50:36.459417+0000","last_became_peered":"2024-03-28T15:50:36.459417+0000","last_unstale":"2024-03-28T15:51:35.353122+0000","last_undegraded":"2024-03-28T15:51:35.353122+0000","last_fullsized":"2024-03-28T15:51:35.353122+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:30:22.022249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"90'9","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351574+0000","last_change":"2024-03-28T15:50:31.426829+0000","last_active":"2024-03-28T15:51:35.351574+0000","last_peered":"2024-03-28T15:51:35.351574+0000","last_clean":"2024-03-28T15:51:35.351574+0000","last_became_active":"2024-03-28T15:50:31.426434+0000","last_became_peered":"2024-03-28T15:50:31.426434+0000","last_unstale":"2024-03-28T15:51:35.351574+0000","last_undegraded":"2024-03-28T15:51:35.351574+0000","last_fullsized":"2024-03-28T15:51:35.351574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:12:47.567766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.404179+0000","last_change":"2024-03-28T15:50:32.427927+0000","last_active":"2024-03-28T15:52:04.404179+0000","last_peered":"2024-03-28T15:52:04.404179+0000","last_clean":"2024-03-28T15:52:04.404179+0000","last_became_active":"2024-03-28T15:50:32.427609+0000","last_became_peered":"2024-03-28T15:50:32.427609+0000","last_unstale":"2024-03-28T15:52:04.404179+0000","last_undegraded":"2024-03-28T15:52:04.404179+0000","last_fullsized":"2024-03-28T15:52:04.404179+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:16:49.120638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351821+0000","last_change":"2024-03-28T15:50:36.460355+0000","last_active":"2024-03-28T15:51:35.351821+0000","last_peered":"2024-03-28T15:51:35.351821+0000","last_clean":"2024-03-28T15:51:35.351821+0000","last_became_active":"2024-03-28T15:50:36.459935+0000","last_became_peered":"2024-03-28T15:50:36.459935+0000","last_unstale":"2024-03-28T15:51:35.351821+0000","last_undegraded":"2024-03-28T15:51:35.351821+0000","last_fullsized":"2024-03-28T15:51:35.351821+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:58:34.895844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"90'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352405+0000","last_change":"2024-03-28T15:50:31.417282+0000","last_active":"2024-03-28T15:51:35.352405+0000","last_peered":"2024-03-28T15:51:35.352405+0000","last_clean":"2024-03-28T15:51:35.352405+0000","last_became_active":"2024-03-28T15:50:31.416852+0000","last_became_peered":"2024-03-28T15:50:31.416852+0000","last_unstale":"2024-03-28T15:51:35.352405+0000","last_undegraded":"2024-03-28T15:51:35.352405+0000","last_fullsized":"2024-03-28T15:51:35.352405+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:26:12.853799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"92'15","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410320+0000","last_change":"2024-03-28T15:50:32.430362+0000","last_active":"2024-03-28T15:52:04.410320+0000","last_peered":"2024-03-28T15:52:04.410320+0000","last_clean":"2024-03-28T15:52:04.410320+0000","last_became_active":"2024-03-28T15:50:32.429999+0000","last_became_peered":"2024-03-28T15:50:32.429999+0000","last_unstale":"2024-03-28T15:52:04.410320+0000","last_undegraded":"2024-03-28T15:52:04.410320+0000","last_fullsized":"2024-03-28T15:52:04.410320+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:32:47.613322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349849+0000","last_change":"2024-03-28T15:50:36.463116+0000","last_active":"2024-03-28T15:51:35.349849+0000","last_peered":"2024-03-28T15:51:35.349849+0000","last_clean":"2024-03-28T15:51:35.349849+0000","last_became_active":"2024-03-28T15:50:36.462705+0000","last_became_peered":"2024-03-28T15:50:36.462705+0000","last_unstale":"2024-03-28T15:51:35.349849+0000","last_undegraded":"2024-03-28T15:51:35.349849+0000","last_fullsized":"2024-03-28T15:51:35.349849+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:59:49.876404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:11.306131+0000","last_change":"2024-03-28T15:50:31.418595+0000","last_active":"2024-03-28T15:52:11.306131+0000","last_peered":"2024-03-28T15:52:11.306131+0000","last_clean":"2024-03-28T15:52:11.306131+0000","last_became_active":"2024-03-28T15:50:31.418185+0000","last_became_peered":"2024-03-28T15:50:31.418185+0000","last_unstale":"2024-03-28T15:52:11.306131+0000","last_undegraded":"2024-03-28T15:52:11.306131+0000","last_fullsized":"2024-03-28T15:52:11.306131+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:40.246760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":77,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407733+0000","last_change":"2024-03-28T15:50:32.431867+0000","last_active":"2024-03-28T15:52:04.407733+0000","last_peered":"2024-03-28T15:52:04.407733+0000","last_clean":"2024-03-28T15:52:04.407733+0000","last_became_active":"2024-03-28T15:50:32.431552+0000","last_became_peered":"2024-03-28T15:50:32.431552+0000","last_unstale":"2024-03-28T15:52:04.407733+0000","last_undegraded":"2024-03-28T15:52:04.407733+0000","last_fullsized":"2024-03-28T15:52:04.407733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:27:20.849087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.935008+0000","last_change":"2024-03-28T15:50:36.466785+0000","last_active":"2024-03-28T15:51:35.935008+0000","last_peered":"2024-03-28T15:51:35.935008+0000","last_clean":"2024-03-28T15:51:35.935008+0000","last_became_active":"2024-03-28T15:50:36.466356+0000","last_became_peered":"2024-03-28T15:50:36.466356+0000","last_unstale":"2024-03-28T15:51:35.935008+0000","last_undegraded":"2024-03-28T15:51:35.935008+0000","last_fullsized":"2024-03-28T15:51:35.935008+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:50:07.453802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"90'17","reported_seq":45,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349090+0000","last_change":"2024-03-28T15:50:31.417956+0000","last_active":"2024-03-28T15:51:35.349090+0000","last_peered":"2024-03-28T15:51:35.349090+0000","last_clean":"2024-03-28T15:51:35.349090+0000","last_became_active":"2024-03-28T15:50:31.417613+0000","last_became_peered":"2024-03-28T15:50:31.417613+0000","last_unstale":"2024-03-28T15:51:35.349090+0000","last_undegraded":"2024-03-28T15:51:35.349090+0000","last_fullsized":"2024-03-28T15:51:35.349090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:08:33.039685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"92'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.413414+0000","last_change":"2024-03-28T15:51:35.936715+0000","last_active":"2024-03-28T15:52:04.413414+0000","last_peered":"2024-03-28T15:52:04.413414+0000","last_clean":"2024-03-28T15:52:04.413414+0000","last_became_active":"2024-03-28T15:51:35.936451+0000","last_became_peered":"2024-03-28T15:51:35.936451+0000","last_unstale":"2024-03-28T15:52:04.413414+0000","last_undegraded":"2024-03-28T15:52:04.413414+0000","last_fullsized":"2024-03-28T15:52:04.413414+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:39:50.654204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352550+0000","last_change":"2024-03-28T15:50:36.463509+0000","last_active":"2024-03-28T15:51:35.352550+0000","last_peered":"2024-03-28T15:51:35.352550+0000","last_clean":"2024-03-28T15:51:35.352550+0000","last_became_active":"2024-03-28T15:50:36.462733+0000","last_became_peered":"2024-03-28T15:50:36.462733+0000","last_unstale":"2024-03-28T15:51:35.352550+0000","last_undegraded":"2024-03-28T15:51:35.352550+0000","last_fullsized":"2024-03-28T15:51:35.352550+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:43:33.737012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"90'2","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349865+0000","last_change":"2024-03-28T15:50:31.420224+0000","last_active":"2024-03-28T15:51:35.349865+0000","last_peered":"2024-03-28T15:51:35.349865+0000","last_clean":"2024-03-28T15:51:35.349865+0000","last_became_active":"2024-03-28T15:50:31.419913+0000","last_became_peered":"2024-03-28T15:50:31.419913+0000","last_unstale":"2024-03-28T15:51:35.349865+0000","last_undegraded":"2024-03-28T15:51:35.349865+0000","last_fullsized":"2024-03-28T15:51:35.349865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:59:49.834297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"92'5","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406280+0000","last_change":"2024-03-28T15:51:35.357722+0000","last_active":"2024-03-28T15:52:04.406280+0000","last_peered":"2024-03-28T15:52:04.406280+0000","last_clean":"2024-03-28T15:52:04.406280+0000","last_became_active":"2024-03-28T15:51:35.357352+0000","last_became_peered":"2024-03-28T15:51:35.357352+0000","last_unstale":"2024-03-28T15:52:04.406280+0000","last_undegraded":"2024-03-28T15:52:04.406280+0000","last_fullsized":"2024-03-28T15:52:04.406280+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:05:13.934522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349311+0000","last_change":"2024-03-28T15:50:36.460744+0000","last_active":"2024-03-28T15:51:35.349311+0000","last_peered":"2024-03-28T15:51:35.349311+0000","last_clean":"2024-03-28T15:51:35.349311+0000","last_became_active":"2024-03-28T15:50:36.460276+0000","last_became_peered":"2024-03-28T15:50:36.460276+0000","last_unstale":"2024-03-28T15:51:35.349311+0000","last_undegraded":"2024-03-28T15:51:35.349311+0000","last_fullsized":"2024-03-28T15:51:35.349311+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:53:12.600154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"90'1","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349390+0000","last_change":"2024-03-28T15:50:31.427718+0000","last_active":"2024-03-28T15:51:35.349390+0000","last_peered":"2024-03-28T15:51:35.349390+0000","last_clean":"2024-03-28T15:51:35.349390+0000","last_became_active":"2024-03-28T15:50:31.427414+0000","last_became_peered":"2024-03-28T15:50:31.427414+0000","last_unstale":"2024-03-28T15:51:35.349390+0000","last_undegraded":"2024-03-28T15:51:35.349390+0000","last_fullsized":"2024-03-28T15:51:35.349390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:29:24.636431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"92'4","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409642+0000","last_change":"2024-03-28T15:50:32.424409+0000","last_active":"2024-03-28T15:52:04.409642+0000","last_peered":"2024-03-28T15:52:04.409642+0000","last_clean":"2024-03-28T15:52:04.409642+0000","last_became_active":"2024-03-28T15:50:32.424108+0000","last_became_peered":"2024-03-28T15:50:32.424108+0000","last_unstale":"2024-03-28T15:52:04.409642+0000","last_undegraded":"2024-03-28T15:52:04.409642+0000","last_fullsized":"2024-03-28T15:52:04.409642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:59:23.953912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934923+0000","last_change":"2024-03-28T15:50:36.463533+0000","last_active":"2024-03-28T15:51:35.934923+0000","last_peered":"2024-03-28T15:51:35.934923+0000","last_clean":"2024-03-28T15:51:35.934923+0000","last_became_active":"2024-03-28T15:50:36.463047+0000","last_became_peered":"2024-03-28T15:50:36.463047+0000","last_unstale":"2024-03-28T15:51:35.934923+0000","last_undegraded":"2024-03-28T15:51:35.934923+0000","last_fullsized":"2024-03-28T15:51:35.934923+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:49:52.072250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"90'5","reported_seq":104,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:10.305196+0000","last_change":"2024-03-28T15:50:31.417073+0000","last_active":"2024-03-28T15:52:10.305196+0000","last_peered":"2024-03-28T15:52:10.305196+0000","last_clean":"2024-03-28T15:52:10.305196+0000","last_became_active":"2024-03-28T15:50:31.416755+0000","last_became_peered":"2024-03-28T15:50:31.416755+0000","last_unstale":"2024-03-28T15:52:10.305196+0000","last_undegraded":"2024-03-28T15:52:10.305196+0000","last_fullsized":"2024-03-28T15:52:10.305196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:45.887894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.398987+0000","last_change":"2024-03-28T15:50:32.424912+0000","last_active":"2024-03-28T15:52:04.398987+0000","last_peered":"2024-03-28T15:52:04.398987+0000","last_clean":"2024-03-28T15:52:04.398987+0000","last_became_active":"2024-03-28T15:50:32.424609+0000","last_became_peered":"2024-03-28T15:50:32.424609+0000","last_unstale":"2024-03-28T15:52:04.398987+0000","last_undegraded":"2024-03-28T15:52:04.398987+0000","last_fullsized":"2024-03-28T15:52:04.398987+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:42:37.539793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934909+0000","last_change":"2024-03-28T15:50:36.463968+0000","last_active":"2024-03-28T15:51:35.934909+0000","last_peered":"2024-03-28T15:51:35.934909+0000","last_clean":"2024-03-28T15:51:35.934909+0000","last_became_active":"2024-03-28T15:50:36.460697+0000","last_became_peered":"2024-03-28T15:50:36.460697+0000","last_unstale":"2024-03-28T15:51:35.934909+0000","last_undegraded":"2024-03-28T15:51:35.934909+0000","last_fullsized":"2024-03-28T15:51:35.934909+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:19:00.726769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403083+0000","last_change":"2024-03-28T15:50:32.419783+0000","last_active":"2024-03-28T15:52:04.403083+0000","last_peered":"2024-03-28T15:52:04.403083+0000","last_clean":"2024-03-28T15:52:04.403083+0000","last_became_active":"2024-03-28T15:50:32.419404+0000","last_became_peered":"2024-03-28T15:50:32.419404+0000","last_unstale":"2024-03-28T15:52:04.403083+0000","last_undegraded":"2024-03-28T15:52:04.403083+0000","last_fullsized":"2024-03-28T15:52:04.403083+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:43:42.437830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"90'5","reported_seq":106,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:11.305679+0000","last_change":"2024-03-28T15:50:31.414630+0000","last_active":"2024-03-28T15:52:11.305679+0000","last_peered":"2024-03-28T15:52:11.305679+0000","last_clean":"2024-03-28T15:52:11.305679+0000","last_became_active":"2024-03-28T15:50:31.414300+0000","last_became_peered":"2024-03-28T15:50:31.414300+0000","last_unstale":"2024-03-28T15:52:11.305679+0000","last_undegraded":"2024-03-28T15:52:11.305679+0000","last_fullsized":"2024-03-28T15:52:11.305679+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:03:48.773649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"90'1","reported_seq":19,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.353277+0000","last_change":"2024-03-28T15:50:36.462352+0000","last_active":"2024-03-28T15:51:35.353277+0000","last_peered":"2024-03-28T15:51:35.353277+0000","last_clean":"2024-03-28T15:51:35.353277+0000","last_became_active":"2024-03-28T15:50:36.462003+0000","last_became_peered":"2024-03-28T15:50:36.462003+0000","last_unstale":"2024-03-28T15:51:35.353277+0000","last_undegraded":"2024-03-28T15:51:35.353277+0000","last_fullsized":"2024-03-28T15:51:35.353277+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:00:42.830355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408134+0000","last_change":"2024-03-28T15:50:32.429086+0000","last_active":"2024-03-28T15:52:04.408134+0000","last_peered":"2024-03-28T15:52:04.408134+0000","last_clean":"2024-03-28T15:52:04.408134+0000","last_became_active":"2024-03-28T15:50:32.427761+0000","last_became_peered":"2024-03-28T15:50:32.427761+0000","last_unstale":"2024-03-28T15:52:04.408134+0000","last_undegraded":"2024-03-28T15:52:04.408134+0000","last_fullsized":"2024-03-28T15:52:04.408134+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:37:53.883733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"90'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349768+0000","last_change":"2024-03-28T15:50:31.417017+0000","last_active":"2024-03-28T15:51:35.349768+0000","last_peered":"2024-03-28T15:51:35.349768+0000","last_clean":"2024-03-28T15:51:35.349768+0000","last_became_active":"2024-03-28T15:50:31.416688+0000","last_became_peered":"2024-03-28T15:50:31.416688+0000","last_unstale":"2024-03-28T15:51:35.349768+0000","last_undegraded":"2024-03-28T15:51:35.349768+0000","last_fullsized":"2024-03-28T15:51:35.349768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:50:39.364544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":360,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352331+0000","last_change":"2024-03-28T15:49:22.209051+0000","last_active":"2024-03-28T15:51:35.352331+0000","last_peered":"2024-03-28T15:51:35.352331+0000","last_clean":"2024-03-28T15:51:35.352331+0000","last_became_active":"2024-03-28T15:49:22.208633+0000","last_became_peered":"2024-03-28T15:49:22.208633+0000","last_unstale":"2024-03-28T15:51:35.352331+0000","last_undegraded":"2024-03-28T15:51:35.352331+0000","last_fullsized":"2024-03-28T15:51:35.352331+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:45:59.266253+0000","last_clean_scrub_stamp":"2024-03-28T15:45:59.266253+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:08:41.194614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351722+0000","last_change":"2024-03-28T15:50:36.463514+0000","last_active":"2024-03-28T15:51:35.351722+0000","last_peered":"2024-03-28T15:51:35.351722+0000","last_clean":"2024-03-28T15:51:35.351722+0000","last_became_active":"2024-03-28T15:50:36.463110+0000","last_became_peered":"2024-03-28T15:50:36.463110+0000","last_unstale":"2024-03-28T15:51:35.351722+0000","last_undegraded":"2024-03-28T15:51:35.351722+0000","last_fullsized":"2024-03-28T15:51:35.351722+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:24:30.481123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402429+0000","last_change":"2024-03-28T15:50:32.427016+0000","last_active":"2024-03-28T15:52:04.402429+0000","last_peered":"2024-03-28T15:52:04.402429+0000","last_clean":"2024-03-28T15:52:04.402429+0000","last_became_active":"2024-03-28T15:50:32.426519+0000","last_became_peered":"2024-03-28T15:50:32.426519+0000","last_unstale":"2024-03-28T15:52:04.402429+0000","last_undegraded":"2024-03-28T15:52:04.402429+0000","last_fullsized":"2024-03-28T15:52:04.402429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:38:27.650290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"90'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352455+0000","last_change":"2024-03-28T15:50:31.420871+0000","last_active":"2024-03-28T15:51:35.352455+0000","last_peered":"2024-03-28T15:51:35.352455+0000","last_clean":"2024-03-28T15:51:35.352455+0000","last_became_active":"2024-03-28T15:50:31.420418+0000","last_became_peered":"2024-03-28T15:50:31.420418+0000","last_unstale":"2024-03-28T15:51:35.352455+0000","last_undegraded":"2024-03-28T15:51:35.352455+0000","last_fullsized":"2024-03-28T15:51:35.352455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:47:57.615539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352424+0000","last_change":"2024-03-28T15:50:36.457759+0000","last_active":"2024-03-28T15:51:35.352424+0000","last_peered":"2024-03-28T15:51:35.352424+0000","last_clean":"2024-03-28T15:51:35.352424+0000","last_became_active":"2024-03-28T15:50:36.457248+0000","last_became_peered":"2024-03-28T15:50:36.457248+0000","last_unstale":"2024-03-28T15:51:35.352424+0000","last_undegraded":"2024-03-28T15:51:35.352424+0000","last_fullsized":"2024-03-28T15:51:35.352424+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:13:30.420180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349512+0000","last_change":"2024-03-28T15:50:31.428270+0000","last_active":"2024-03-28T15:51:35.349512+0000","last_peered":"2024-03-28T15:51:35.349512+0000","last_clean":"2024-03-28T15:51:35.349512+0000","last_became_active":"2024-03-28T15:50:31.427888+0000","last_became_peered":"2024-03-28T15:50:31.427888+0000","last_unstale":"2024-03-28T15:51:35.349512+0000","last_undegraded":"2024-03-28T15:51:35.349512+0000","last_fullsized":"2024-03-28T15:51:35.349512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:48:51.226461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.401429+0000","last_change":"2024-03-28T15:50:32.416631+0000","last_active":"2024-03-28T15:52:04.401429+0000","last_peered":"2024-03-28T15:52:04.401429+0000","last_clean":"2024-03-28T15:52:04.401429+0000","last_became_active":"2024-03-28T15:50:32.416300+0000","last_became_peered":"2024-03-28T15:50:32.416300+0000","last_unstale":"2024-03-28T15:52:04.401429+0000","last_undegraded":"2024-03-28T15:52:04.401429+0000","last_fullsized":"2024-03-28T15:52:04.401429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:58:19.158562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.853207+0000","last_change":"2024-03-28T15:50:36.465991+0000","last_active":"2024-03-28T15:51:35.853207+0000","last_peered":"2024-03-28T15:51:35.853207+0000","last_clean":"2024-03-28T15:51:35.853207+0000","last_became_active":"2024-03-28T15:50:36.465506+0000","last_became_peered":"2024-03-28T15:50:36.465506+0000","last_unstale":"2024-03-28T15:51:35.853207+0000","last_undegraded":"2024-03-28T15:51:35.853207+0000","last_fullsized":"2024-03-28T15:51:35.853207+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:35:22.889068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"92'13","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349979+0000","last_change":"2024-03-28T15:50:31.423894+0000","last_active":"2024-03-28T15:51:35.349979+0000","last_peered":"2024-03-28T15:51:35.349979+0000","last_clean":"2024-03-28T15:51:35.349979+0000","last_became_active":"2024-03-28T15:50:31.423562+0000","last_became_peered":"2024-03-28T15:50:31.423562+0000","last_unstale":"2024-03-28T15:51:35.349979+0000","last_undegraded":"2024-03-28T15:51:35.349979+0000","last_fullsized":"2024-03-28T15:51:35.349979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:35:55.065632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"92'5","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.409951+0000","last_change":"2024-03-28T15:50:32.435132+0000","last_active":"2024-03-28T15:52:04.409951+0000","last_peered":"2024-03-28T15:52:04.409951+0000","last_clean":"2024-03-28T15:52:04.409951+0000","last_became_active":"2024-03-28T15:50:32.434523+0000","last_became_peered":"2024-03-28T15:50:32.434523+0000","last_unstale":"2024-03-28T15:52:04.409951+0000","last_undegraded":"2024-03-28T15:52:04.409951+0000","last_fullsized":"2024-03-28T15:52:04.409951+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:21:35.363388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351262+0000","last_change":"2024-03-28T15:50:36.453501+0000","last_active":"2024-03-28T15:51:35.351262+0000","last_peered":"2024-03-28T15:51:35.351262+0000","last_clean":"2024-03-28T15:51:35.351262+0000","last_became_active":"2024-03-28T15:50:36.452121+0000","last_became_peered":"2024-03-28T15:50:36.452121+0000","last_unstale":"2024-03-28T15:51:35.351262+0000","last_undegraded":"2024-03-28T15:51:35.351262+0000","last_fullsized":"2024-03-28T15:51:35.351262+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:48:33.354233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"90'7","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.350253+0000","last_change":"2024-03-28T15:50:31.425645+0000","last_active":"2024-03-28T15:51:35.350253+0000","last_peered":"2024-03-28T15:51:35.350253+0000","last_clean":"2024-03-28T15:51:35.350253+0000","last_became_active":"2024-03-28T15:50:31.425332+0000","last_became_peered":"2024-03-28T15:50:31.425332+0000","last_unstale":"2024-03-28T15:51:35.350253+0000","last_undegraded":"2024-03-28T15:51:35.350253+0000","last_fullsized":"2024-03-28T15:51:35.350253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:09:16.221936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410597+0000","last_change":"2024-03-28T15:50:32.432602+0000","last_active":"2024-03-28T15:52:04.410597+0000","last_peered":"2024-03-28T15:52:04.410597+0000","last_clean":"2024-03-28T15:52:04.410597+0000","last_became_active":"2024-03-28T15:50:32.432303+0000","last_became_peered":"2024-03-28T15:50:32.432303+0000","last_unstale":"2024-03-28T15:52:04.410597+0000","last_undegraded":"2024-03-28T15:52:04.410597+0000","last_fullsized":"2024-03-28T15:52:04.410597+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:54:27.524582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.566352+0000","last_change":"2024-03-28T15:51:35.566352+0000","last_active":"2024-03-28T15:51:35.566352+0000","last_peered":"2024-03-28T15:51:35.566352+0000","last_clean":"2024-03-28T15:51:35.566352+0000","last_became_active":"2024-03-28T15:51:35.565829+0000","last_became_peered":"2024-03-28T15:51:35.565829+0000","last_unstale":"2024-03-28T15:51:35.566352+0000","last_undegraded":"2024-03-28T15:51:35.566352+0000","last_fullsized":"2024-03-28T15:51:35.566352+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:00:19.067884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"90'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.718257+0000","last_change":"2024-03-28T15:50:31.418403+0000","last_active":"2024-03-28T15:51:35.718257+0000","last_peered":"2024-03-28T15:51:35.718257+0000","last_clean":"2024-03-28T15:51:35.718257+0000","last_became_active":"2024-03-28T15:50:31.418102+0000","last_became_peered":"2024-03-28T15:50:31.418102+0000","last_unstale":"2024-03-28T15:51:35.718257+0000","last_undegraded":"2024-03-28T15:51:35.718257+0000","last_fullsized":"2024-03-28T15:51:35.718257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:01:56.175404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.370732+0000","last_change":"2024-03-28T15:50:32.417331+0000","last_active":"2024-03-28T15:52:04.370732+0000","last_peered":"2024-03-28T15:52:04.370732+0000","last_clean":"2024-03-28T15:52:04.370732+0000","last_became_active":"2024-03-28T15:50:32.417026+0000","last_became_peered":"2024-03-28T15:50:32.417026+0000","last_unstale":"2024-03-28T15:52:04.370732+0000","last_undegraded":"2024-03-28T15:52:04.370732+0000","last_fullsized":"2024-03-28T15:52:04.370732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:09:24.081575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349990+0000","last_change":"2024-03-28T15:50:36.454877+0000","last_active":"2024-03-28T15:51:35.349990+0000","last_peered":"2024-03-28T15:51:35.349990+0000","last_clean":"2024-03-28T15:51:35.349990+0000","last_became_active":"2024-03-28T15:50:36.453802+0000","last_became_peered":"2024-03-28T15:50:36.453802+0000","last_unstale":"2024-03-28T15:51:35.349990+0000","last_undegraded":"2024-03-28T15:51:35.349990+0000","last_fullsized":"2024-03-28T15:51:35.349990+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:52:56.479558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"90'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.852685+0000","last_change":"2024-03-28T15:50:31.425672+0000","last_active":"2024-03-28T15:51:35.852685+0000","last_peered":"2024-03-28T15:51:35.852685+0000","last_clean":"2024-03-28T15:51:35.852685+0000","last_became_active":"2024-03-28T15:50:31.425324+0000","last_became_peered":"2024-03-28T15:50:31.425324+0000","last_unstale":"2024-03-28T15:51:35.852685+0000","last_undegraded":"2024-03-28T15:51:35.852685+0000","last_fullsized":"2024-03-28T15:51:35.852685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:48:46.380962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408297+0000","last_change":"2024-03-28T15:50:32.424831+0000","last_active":"2024-03-28T15:52:04.408297+0000","last_peered":"2024-03-28T15:52:04.408297+0000","last_clean":"2024-03-28T15:52:04.408297+0000","last_became_active":"2024-03-28T15:50:32.423678+0000","last_became_peered":"2024-03-28T15:50:32.423678+0000","last_unstale":"2024-03-28T15:52:04.408297+0000","last_undegraded":"2024-03-28T15:52:04.408297+0000","last_fullsized":"2024-03-28T15:52:04.408297+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:35:06.895634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"90'1","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.570333+0000","last_change":"2024-03-28T15:51:35.570333+0000","last_active":"2024-03-28T15:51:35.570333+0000","last_peered":"2024-03-28T15:51:35.570333+0000","last_clean":"2024-03-28T15:51:35.570333+0000","last_became_active":"2024-03-28T15:51:35.566494+0000","last_became_peered":"2024-03-28T15:51:35.566494+0000","last_unstale":"2024-03-28T15:51:35.570333+0000","last_undegraded":"2024-03-28T15:51:35.570333+0000","last_fullsized":"2024-03-28T15:51:35.570333+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:47:34.617415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351259+0000","last_change":"2024-03-28T15:50:31.430248+0000","last_active":"2024-03-28T15:51:35.351259+0000","last_peered":"2024-03-28T15:51:35.351259+0000","last_clean":"2024-03-28T15:51:35.351259+0000","last_became_active":"2024-03-28T15:50:31.429964+0000","last_became_peered":"2024-03-28T15:50:31.429964+0000","last_unstale":"2024-03-28T15:51:35.351259+0000","last_undegraded":"2024-03-28T15:51:35.351259+0000","last_fullsized":"2024-03-28T15:51:35.351259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:10:25.193102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.408504+0000","last_change":"2024-03-28T15:50:32.431687+0000","last_active":"2024-03-28T15:52:04.408504+0000","last_peered":"2024-03-28T15:52:04.408504+0000","last_clean":"2024-03-28T15:52:04.408504+0000","last_became_active":"2024-03-28T15:50:32.431295+0000","last_became_peered":"2024-03-28T15:50:32.431295+0000","last_unstale":"2024-03-28T15:52:04.408504+0000","last_undegraded":"2024-03-28T15:52:04.408504+0000","last_fullsized":"2024-03-28T15:52:04.408504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:46:12.055577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.350120+0000","last_change":"2024-03-28T15:50:36.459198+0000","last_active":"2024-03-28T15:51:35.350120+0000","last_peered":"2024-03-28T15:51:35.350120+0000","last_clean":"2024-03-28T15:51:35.350120+0000","last_became_active":"2024-03-28T15:50:36.458815+0000","last_became_peered":"2024-03-28T15:50:36.458815+0000","last_unstale":"2024-03-28T15:51:35.350120+0000","last_undegraded":"2024-03-28T15:51:35.350120+0000","last_fullsized":"2024-03-28T15:51:35.350120+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:55:19.302434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"90'4","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.349163+0000","last_change":"2024-03-28T15:50:31.421319+0000","last_active":"2024-03-28T15:51:35.349163+0000","last_peered":"2024-03-28T15:51:35.349163+0000","last_clean":"2024-03-28T15:51:35.349163+0000","last_became_active":"2024-03-28T15:50:31.420399+0000","last_became_peered":"2024-03-28T15:50:31.420399+0000","last_unstale":"2024-03-28T15:51:35.349163+0000","last_undegraded":"2024-03-28T15:51:35.349163+0000","last_fullsized":"2024-03-28T15:51:35.349163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:48:45.594801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"92'5","reported_seq":29,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.414233+0000","last_change":"2024-03-28T15:50:32.436155+0000","last_active":"2024-03-28T15:52:04.414233+0000","last_peered":"2024-03-28T15:52:04.414233+0000","last_clean":"2024-03-28T15:52:04.414233+0000","last_became_active":"2024-03-28T15:50:32.435453+0000","last_became_peered":"2024-03-28T15:50:32.435453+0000","last_unstale":"2024-03-28T15:52:04.414233+0000","last_undegraded":"2024-03-28T15:52:04.414233+0000","last_fullsized":"2024-03-28T15:52:04.414233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:57:44.159744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.718568+0000","last_change":"2024-03-28T15:50:36.456103+0000","last_active":"2024-03-28T15:51:35.718568+0000","last_peered":"2024-03-28T15:51:35.718568+0000","last_clean":"2024-03-28T15:51:35.718568+0000","last_became_active":"2024-03-28T15:50:36.455724+0000","last_became_peered":"2024-03-28T15:50:36.455724+0000","last_unstale":"2024-03-28T15:51:35.718568+0000","last_undegraded":"2024-03-28T15:51:35.718568+0000","last_fullsized":"2024-03-28T15:51:35.718568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:38:30.874245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"90'5","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:11.297140+0000","last_change":"2024-03-28T15:50:31.423648+0000","last_active":"2024-03-28T15:52:11.297140+0000","last_peered":"2024-03-28T15:52:11.297140+0000","last_clean":"2024-03-28T15:52:11.297140+0000","last_became_active":"2024-03-28T15:50:31.423343+0000","last_became_peered":"2024-03-28T15:50:31.423343+0000","last_unstale":"2024-03-28T15:52:11.297140+0000","last_undegraded":"2024-03-28T15:52:11.297140+0000","last_fullsized":"2024-03-28T15:52:11.297140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:31:22.475804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"92'7","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.417440+0000","last_change":"2024-03-28T15:50:32.432943+0000","last_active":"2024-03-28T15:52:04.417440+0000","last_peered":"2024-03-28T15:52:04.417440+0000","last_clean":"2024-03-28T15:52:04.417440+0000","last_became_active":"2024-03-28T15:50:32.432535+0000","last_became_peered":"2024-03-28T15:50:32.432535+0000","last_unstale":"2024-03-28T15:52:04.417440+0000","last_undegraded":"2024-03-28T15:52:04.417440+0000","last_fullsized":"2024-03-28T15:52:04.417440+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T21:20:17.553547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":25,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.719765+0000","last_change":"2024-03-28T15:51:35.719765+0000","last_active":"2024-03-28T15:51:35.719765+0000","last_peered":"2024-03-28T15:51:35.719765+0000","last_clean":"2024-03-28T15:51:35.719765+0000","last_became_active":"2024-03-28T15:51:35.719380+0000","last_became_peered":"2024-03-28T15:51:35.719380+0000","last_unstale":"2024-03-28T15:51:35.719765+0000","last_undegraded":"2024-03-28T15:51:35.719765+0000","last_fullsized":"2024-03-28T15:51:35.719765+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T17:28:56.441770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"90'3","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.352914+0000","last_change":"2024-03-28T15:50:31.419043+0000","last_active":"2024-03-28T15:51:35.352914+0000","last_peered":"2024-03-28T15:51:35.352914+0000","last_clean":"2024-03-28T15:51:35.352914+0000","last_became_active":"2024-03-28T15:50:31.418722+0000","last_became_peered":"2024-03-28T15:50:31.418722+0000","last_unstale":"2024-03-28T15:51:35.352914+0000","last_undegraded":"2024-03-28T15:51:35.352914+0000","last_fullsized":"2024-03-28T15:51:35.352914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T00:01:10.512442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"92'6","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406546+0000","last_change":"2024-03-28T15:51:35.356071+0000","last_active":"2024-03-28T15:52:04.406546+0000","last_peered":"2024-03-28T15:52:04.406546+0000","last_clean":"2024-03-28T15:52:04.406546+0000","last_became_active":"2024-03-28T15:51:35.355695+0000","last_became_peered":"2024-03-28T15:51:35.355695+0000","last_unstale":"2024-03-28T15:52:04.406546+0000","last_undegraded":"2024-03-28T15:52:04.406546+0000","last_fullsized":"2024-03-28T15:52:04.406546+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T23:27:10.535552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.351755+0000","last_change":"2024-03-28T15:50:36.455924+0000","last_active":"2024-03-28T15:51:35.351755+0000","last_peered":"2024-03-28T15:51:35.351755+0000","last_clean":"2024-03-28T15:51:35.351755+0000","last_became_active":"2024-03-28T15:50:36.455511+0000","last_became_peered":"2024-03-28T15:50:36.455511+0000","last_unstale":"2024-03-28T15:51:35.351755+0000","last_undegraded":"2024-03-28T15:51:35.351755+0000","last_fullsized":"2024-03-28T15:51:35.351755+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":88,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:35.426903+0000","last_clean_scrub_stamp":"2024-03-28T15:50:35.426903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:48.039313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"90'5","reported_seq":31,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:51:35.934572+0000","last_change":"2024-03-28T15:50:31.418393+0000","last_active":"2024-03-28T15:51:35.934572+0000","last_peered":"2024-03-28T15:51:35.934572+0000","last_clean":"2024-03-28T15:51:35.934572+0000","last_became_active":"2024-03-28T15:50:31.418086+0000","last_became_peered":"2024-03-28T15:50:31.418086+0000","last_unstale":"2024-03-28T15:51:35.934572+0000","last_undegraded":"2024-03-28T15:51:35.934572+0000","last_fullsized":"2024-03-28T15:51:35.934572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:30.387040+0000","last_clean_scrub_stamp":"2024-03-28T15:50:30.387040+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:43:44.882002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.411962+0000","last_change":"2024-03-28T15:50:32.429008+0000","last_active":"2024-03-28T15:52:04.411962+0000","last_peered":"2024-03-28T15:52:04.411962+0000","last_clean":"2024-03-28T15:52:04.411962+0000","last_became_active":"2024-03-28T15:50:32.428398+0000","last_became_peered":"2024-03-28T15:50:32.428398+0000","last_unstale":"2024-03-28T15:52:04.411962+0000","last_undegraded":"2024-03-28T15:52:04.411962+0000","last_fullsized":"2024-03-28T15:52:04.411962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:10:50.908576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"92'9","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.404183+0000","last_change":"2024-03-28T15:50:32.424785+0000","last_active":"2024-03-28T15:52:04.404183+0000","last_peered":"2024-03-28T15:52:04.404183+0000","last_clean":"2024-03-28T15:52:04.404183+0000","last_became_active":"2024-03-28T15:50:32.424134+0000","last_became_peered":"2024-03-28T15:50:32.424134+0000","last_unstale":"2024-03-28T15:52:04.404183+0000","last_undegraded":"2024-03-28T15:52:04.404183+0000","last_fullsized":"2024-03-28T15:52:04.404183+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T03:12:20.842423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"92'12","reported_seq":43,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.410620+0000","last_change":"2024-03-28T15:51:35.356185+0000","last_active":"2024-03-28T15:52:04.410620+0000","last_peered":"2024-03-28T15:52:04.410620+0000","last_clean":"2024-03-28T15:52:04.410620+0000","last_became_active":"2024-03-28T15:51:35.355709+0000","last_became_peered":"2024-03-28T15:51:35.355709+0000","last_unstale":"2024-03-28T15:52:04.410620+0000","last_undegraded":"2024-03-28T15:52:04.410620+0000","last_fullsized":"2024-03-28T15:52:04.410620+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:02:33.178070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.393753+0000","last_change":"2024-03-28T15:50:32.425448+0000","last_active":"2024-03-28T15:52:04.393753+0000","last_peered":"2024-03-28T15:52:04.393753+0000","last_clean":"2024-03-28T15:52:04.393753+0000","last_became_active":"2024-03-28T15:50:32.425148+0000","last_became_peered":"2024-03-28T15:50:32.425148+0000","last_unstale":"2024-03-28T15:52:04.393753+0000","last_undegraded":"2024-03-28T15:52:04.393753+0000","last_fullsized":"2024-03-28T15:52:04.393753+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T20:53:31.399253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"92'7","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.411001+0000","last_change":"2024-03-28T15:50:32.433610+0000","last_active":"2024-03-28T15:52:04.411001+0000","last_peered":"2024-03-28T15:52:04.411001+0000","last_clean":"2024-03-28T15:52:04.411001+0000","last_became_active":"2024-03-28T15:50:32.433096+0000","last_became_peered":"2024-03-28T15:50:32.433096+0000","last_unstale":"2024-03-28T15:52:04.411001+0000","last_undegraded":"2024-03-28T15:52:04.411001+0000","last_fullsized":"2024-03-28T15:52:04.411001+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T01:03:20.479797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"92'4","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.402852+0000","last_change":"2024-03-28T15:50:32.428195+0000","last_active":"2024-03-28T15:52:04.402852+0000","last_peered":"2024-03-28T15:52:04.402852+0000","last_clean":"2024-03-28T15:52:04.402852+0000","last_became_active":"2024-03-28T15:50:32.427877+0000","last_became_peered":"2024-03-28T15:50:32.427877+0000","last_unstale":"2024-03-28T15:52:04.402852+0000","last_undegraded":"2024-03-28T15:52:04.402852+0000","last_fullsized":"2024-03-28T15:52:04.402852+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T19:29:56.998688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406562+0000","last_change":"2024-03-28T15:50:32.420177+0000","last_active":"2024-03-28T15:52:04.406562+0000","last_peered":"2024-03-28T15:52:04.406562+0000","last_clean":"2024-03-28T15:52:04.406562+0000","last_became_active":"2024-03-28T15:50:32.419805+0000","last_became_peered":"2024-03-28T15:50:32.419805+0000","last_unstale":"2024-03-28T15:52:04.406562+0000","last_undegraded":"2024-03-28T15:52:04.406562+0000","last_fullsized":"2024-03-28T15:52:04.406562+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:04:31.110858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"92'15","reported_seq":40,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403874+0000","last_change":"2024-03-28T15:50:32.417500+0000","last_active":"2024-03-28T15:52:04.403874+0000","last_peered":"2024-03-28T15:52:04.403874+0000","last_clean":"2024-03-28T15:52:04.403874+0000","last_became_active":"2024-03-28T15:50:32.416843+0000","last_became_peered":"2024-03-28T15:50:32.416843+0000","last_unstale":"2024-03-28T15:52:04.403874+0000","last_undegraded":"2024-03-28T15:52:04.403874+0000","last_fullsized":"2024-03-28T15:52:04.403874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:18:03.656816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"92'2","reported_seq":27,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.403845+0000","last_change":"2024-03-28T15:50:32.428655+0000","last_active":"2024-03-28T15:52:04.403845+0000","last_peered":"2024-03-28T15:52:04.403845+0000","last_clean":"2024-03-28T15:52:04.403845+0000","last_became_active":"2024-03-28T15:50:32.428297+0000","last_became_peered":"2024-03-28T15:50:32.428297+0000","last_unstale":"2024-03-28T15:52:04.403845+0000","last_undegraded":"2024-03-28T15:52:04.403845+0000","last_fullsized":"2024-03-28T15:52:04.403845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T15:55:32.512495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"92'6","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.405148+0000","last_change":"2024-03-28T15:50:32.422012+0000","last_active":"2024-03-28T15:52:04.405148+0000","last_peered":"2024-03-28T15:52:04.405148+0000","last_clean":"2024-03-28T15:52:04.405148+0000","last_became_active":"2024-03-28T15:50:32.421465+0000","last_became_peered":"2024-03-28T15:50:32.421465+0000","last_unstale":"2024-03-28T15:52:04.405148+0000","last_undegraded":"2024-03-28T15:52:04.405148+0000","last_fullsized":"2024-03-28T15:52:04.405148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-30T02:41:35.235917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"92'8","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.407293+0000","last_change":"2024-03-28T15:50:32.426624+0000","last_active":"2024-03-28T15:52:04.407293+0000","last_peered":"2024-03-28T15:52:04.407293+0000","last_clean":"2024-03-28T15:52:04.407293+0000","last_became_active":"2024-03-28T15:50:32.425307+0000","last_became_peered":"2024-03-28T15:50:32.425307+0000","last_unstale":"2024-03-28T15:52:04.407293+0000","last_undegraded":"2024-03-28T15:52:04.407293+0000","last_fullsized":"2024-03-28T15:52:04.407293+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:16:05.332057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"92'10","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.420108+0000","last_change":"2024-03-28T15:50:32.430274+0000","last_active":"2024-03-28T15:52:04.420108+0000","last_peered":"2024-03-28T15:52:04.420108+0000","last_clean":"2024-03-28T15:52:04.420108+0000","last_became_active":"2024-03-28T15:50:32.429147+0000","last_became_peered":"2024-03-28T15:50:32.429147+0000","last_unstale":"2024-03-28T15:52:04.420108+0000","last_undegraded":"2024-03-28T15:52:04.420108+0000","last_fullsized":"2024-03-28T15:52:04.420108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T16:09:19.135249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"92'11","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.399624+0000","last_change":"2024-03-28T15:50:32.434933+0000","last_active":"2024-03-28T15:52:04.399624+0000","last_peered":"2024-03-28T15:52:04.399624+0000","last_clean":"2024-03-28T15:52:04.399624+0000","last_became_active":"2024-03-28T15:50:32.434612+0000","last_became_peered":"2024-03-28T15:50:32.434612+0000","last_unstale":"2024-03-28T15:52:04.399624+0000","last_undegraded":"2024-03-28T15:52:04.399624+0000","last_fullsized":"2024-03-28T15:52:04.399624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T22:20:41.236770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"92'8","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-28T15:52:04.406746+0000","last_change":"2024-03-28T15:50:32.426181+0000","last_active":"2024-03-28T15:52:04.406746+0000","last_peered":"2024-03-28T15:52:04.406746+0000","last_clean":"2024-03-28T15:52:04.406746+0000","last_became_active":"2024-03-28T15:50:32.425866+0000","last_became_peered":"2024-03-28T15:50:32.425866+0000","last_unstale":"2024-03-28T15:52:04.406746+0000","last_undegraded":"2024-03-28T15:52:04.406746+0000","last_fullsized":"2024-03-28T15:52:04.406746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T15:50:31.389364+0000","last_clean_scrub_stamp":"2024-03-28T15:50:31.389364+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-29T18:25:42.519070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":250168,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66074,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":764,"num_read_kb":454,"num_write":246,"num_write_kb":140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":504484,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32312,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712840,"statfs":{"total":95995035648,"available":95961948160,"internally_reserved":0,"allocated":1163264,"data_stored":1154516,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":504414,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":504792,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299446,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32236,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712916,"statfs":{"total":95995035648,"available":95962025984,"internally_reserved":0,"allocated":1085440,"data_stored":1101430,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462971,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":499918,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36388,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708764,"statfs":{"total":95995035648,"available":95957774336,"internally_reserved":0,"allocated":1077248,"data_stored":560068,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1036288,"data_stored":501374,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":560820,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1024000,"data_stored":499626,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888133,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":503992,"data_compressed":20647,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724608,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36404,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708748,"statfs":{"total":95995035648,"available":95957757952,"internally_reserved":0,"allocated":1093632,"data_stored":1097012,"data_compressed":57086,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34804,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36439,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-28T15:52:16.030 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-28T15:52:16.030 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T15:52:16.030 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-28T15:52:16.030 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-28T15:52:16.283 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:15 smithi117 ceph-mon[36002]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:16.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:15 smithi117 ceph-mon[36002]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:15 smithi120 ceph-mon[36826]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:16.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:15 smithi120 ceph-mon[36826]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:16.404 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:15 smithi023 ceph-mon[32448]: pgmap v342: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:16.405 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:15 smithi023 ceph-mon[32448]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-28T15:52:17.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2972020466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 msgr2=0x7f2dd410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:17.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2972020466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd410df10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00b0a0 tx=0x7f2dcc00b3b0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2972020466 shutdown_connections 2024-03-28T15:52:17.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2972020466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd410df10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2972020466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 0x7f2dd41074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2972020466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 0x7f2dd41014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2972020466 >> 172.21.15.23:0/2972020466 conn(0x7f2dd4077fc0 msgr2=0x7f2dd40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2972020466 shutdown_connections 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2972020466 wait complete. 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.971+0000 7f2dd9b0e700 1 Processor -- start 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 -- start start 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 0x7f2dd419e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 0x7f2dd41986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd4111370 con 0x7f2dd4101a30 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41111f0 con 0x7f2dd4101090 2024-03-28T15:52:17.975 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd9b0e700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd41114f0 con 0x7f2dd41070d0 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:51942/0 (socket says 172.21.15.23:51942) 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 -- 172.21.15.23:0/2729348961 learned_addr learned my addr 172.21.15.23:0/2729348961 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd37fe700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 0x7f2dd419e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd3fff700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 0x7f2dd41986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 -- 172.21.15.23:0/2729348961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 msgr2=0x7f2dd41986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 0x7f2dd41986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 -- 172.21.15.23:0/2729348961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 msgr2=0x7f2dd419e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 0x7f2dd419e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.972+0000 7f2dd2ffd700 1 -- 172.21.15.23:0/2729348961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd4198fd0 con 0x7f2dd4101a30 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd2ffd700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800eb30 tx=0x7f2dc800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc800cbd0 con 0x7f2dd4101a30 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd41992c0 con 0x7f2dd4101a30 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2dc8004500 con 0x7f2dd4101a30 2024-03-28T15:52:17.976 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dc8010430 con 0x7f2dd4101a30 2024-03-28T15:52:17.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.973+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd41a55a0 con 0x7f2dd4101a30 2024-03-28T15:52:17.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.974+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd4102780 con 0x7f2dd4101a30 2024-03-28T15:52:17.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.975+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2dc8003f00 con 0x7f2dd4101a30 2024-03-28T15:52:17.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.976+0000 7f2dd0ff9700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 0x7f2dbc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:17.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.976+0000 7f2dd37fe700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 0x7f2dbc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:17.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.976+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2dc80cf120 con 0x7f2dd4101a30 2024-03-28T15:52:17.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.976+0000 7f2dd37fe700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 0x7f2dbc0a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009be0 tx=0x7f2dc4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:17.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:17.980+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2dc801f340 con 0x7f2dd4101a30 2024-03-28T15:52:18.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.235+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2dd4068e50 con 0x7f2dd4101a30 2024-03-28T15:52:18.237 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:17 smithi023 ceph-mon[32448]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:18.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.236+0000 7f2dd0ff9700 1 -- 172.21.15.23:0/2729348961 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2dc809bef0 con 0x7f2dd4101a30 2024-03-28T15:52:18.238 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:18.238 INFO:teuthology.orchestra.run.smithi023.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 msgr2=0x7f2dbc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 0x7f2dbc0a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2dc4009be0 tx=0x7f2dc4009260 comp rx=0 tx=0).stop 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 msgr2=0x7f2dd419eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2dc800eb30 tx=0x7f2dc800ee40 comp rx=0 tx=0).stop 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 shutdown_connections 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2dbc0a2910 0x7f2dbc0a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2dd41070d0 0x7f2dd41986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2dd4101090 0x7f2dd419e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:18.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 --2- 172.21.15.23:0/2729348961 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2dd4101a30 0x7f2dd419eb60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:18.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 >> 172.21.15.23:0/2729348961 conn(0x7f2dd4077fc0 msgr2=0x7f2dd4100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:18.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 shutdown_connections 2024-03-28T15:52:18.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:18.239+0000 7f2dd9b0e700 1 -- 172.21.15.23:0/2729348961 wait complete. 2024-03-28T15:52:18.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:17 smithi117 ceph-mon[36002]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:18.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:17 smithi120 ceph-mon[36826]: pgmap v343: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-28T15:52:18.947 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-28T15:52:18.948 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- 172.21.15.23:0/3706294909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f8102170 msgr2=0x7fe9f8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 --2- 172.21.15.23:0/3706294909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f8102170 0x7fe9f8108a00 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe9f0009ca0 tx=0x7fe9f002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- 172.21.15.23:0/3706294909 shutdown_connections 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 --2- 172.21.15.23:0/3706294909 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f8102170 0x7fe9f8108a00 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 --2- 172.21.15.23:0/3706294909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 0x7fe9f8101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 --2- 172.21.15.23:0/3706294909 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f81005c0 0x7fe9f81009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- 172.21.15.23:0/3706294909 >> 172.21.15.23:0/3706294909 conn(0x7fe9f80fbb30 msgr2=0x7fe9f80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- 172.21.15.23:0/3706294909 shutdown_connections 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- 172.21.15.23:0/3706294909 wait complete. 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 Processor -- start 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.054+0000 7fea00475640 1 -- start start 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 0x7fe9f819c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f8102170 0x7fe9f819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f810c240 con 0x7fe9f81005c0 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f810c0c0 con 0x7fe9f81017b0 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f810c3c0 con 0x7fe9f8102170 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.058 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:57992/0 (socket says 172.21.15.23:57992) 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 -- 172.21.15.23:0/1284067212 learned_addr learned my addr 172.21.15.23:0/1284067212 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fda10640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 0x7fe9f819c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fea12640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f8102170 0x7fe9f819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 -- 172.21.15.23:0/1284067212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f8102170 msgr2=0x7fe9f819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f8102170 0x7fe9f819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 -- 172.21.15.23:0/1284067212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 msgr2=0x7fe9f819c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 0x7fe9f819c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 -- 172.21.15.23:0/1284067212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f819f5e0 con 0x7fe9f81005c0 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9fe211640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400e9c0 tx=0x7fe9f400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400cd80 con 0x7fe9f81005c0 2024-03-28T15:52:19.059 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9f40035f0 con 0x7fe9f81005c0 2024-03-28T15:52:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f819f8d0 con 0x7fe9f81005c0 2024-03-28T15:52:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4010690 con 0x7fe9f81005c0 2024-03-28T15:52:19.060 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.055+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f819fdd0 con 0x7fe9f81005c0 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.056+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f81078e0 con 0x7fe9f81005c0 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.057+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe9f40037a0 con 0x7fe9f81005c0 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.058+0000 7fe9ef7fe640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 0x7fe9e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.058+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe9f40cf120 con 0x7fe9f81005c0 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.058+0000 7fe9fda10640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 0x7fe9e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.058+0000 7fe9fda10640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 0x7fe9e40a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8069100 tx=0x7fe9e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.063 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.062+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe9f409be60 con 0x7fe9f81005c0 2024-03-28T15:52:19.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:18 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2729348961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T15:52:19.222 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:18 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2729348961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T15:52:19.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:18 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2729348961' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-28T15:52:19.355 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.354+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9f80622a0 con 0x7fe9f81005c0 2024-03-28T15:52:19.357 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.356+0000 7fe9ef7fe640 1 -- 172.21.15.23:0/1284067212 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24622 (secure 0 0 0) 0x7fe9f409b760 con 0x7fe9f81005c0 2024-03-28T15:52:19.357 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:19.357 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:51:35.345652+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 msgr2=0x7fe9e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 0x7fe9e40a4cc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8069100 tx=0x7fe9e8005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 msgr2=0x7fe9f8069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400e9c0 tx=0x7fe9f400ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 shutdown_connections 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe9e40a2800 0x7fe9e40a4cc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe9f8102170 0x7fe9f819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9f81017b0 0x7fe9f819c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 --2- 172.21.15.23:0/1284067212 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe9f81005c0 0x7fe9f8069390 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 >> 172.21.15.23:0/1284067212 conn(0x7fe9f80fbb30 msgr2=0x7fe9f8104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:19.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 shutdown_connections 2024-03-28T15:52:19.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.358+0000 7fea00475640 1 -- 172.21.15.23:0/1284067212 wait complete. 2024-03-28T15:52:19.371 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:52:19.371 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 -- 172.21.15.23:0/4248055305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68101790 msgr2=0x7f8c68101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/4248055305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68101790 0x7f8c68101c10 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c009900 tx=0x7f8c5c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 -- 172.21.15.23:0/4248055305 shutdown_connections 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/4248055305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68102150 0x7f8c681089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/4248055305 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68101790 0x7f8c68101c10 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/4248055305 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c681009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.475+0000 7f8c6e39d640 1 -- 172.21.15.23:0/4248055305 >> 172.21.15.23:0/4248055305 conn(0x7f8c680fbb30 msgr2=0x7f8c680fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- 172.21.15.23:0/4248055305 shutdown_connections 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- 172.21.15.23:0/4248055305 wait complete. 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 Processor -- start 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- start start 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 0x7f8c6819c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.481 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6810c230 con 0x7f8c68102150 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6810c0b0 con 0x7f8c68101790 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.476+0000 7f8c6e39d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6810c3b0 con 0x7f8c681005a0 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6c93a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6c93a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58002/0 (socket says 172.21.15.23:58002) 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:44766/0 (socket says 172.21.15.23:44766) 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c677fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 0x7f8c6819c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6c93a640 1 -- 172.21.15.23:0/316985536 learned_addr learned my addr 172.21.15.23:0/316985536 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 -- 172.21.15.23:0/316985536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 msgr2=0x7f8c6819c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 0x7f8c6819c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 -- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 msgr2=0x7f8c6819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.482 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 -- 172.21.15.23:0/316985536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6819f650 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6c93a640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c677fe640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 0x7f8c6819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c67fff640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800e9c0 tx=0x7f8c5800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5800ccc0 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6819f8e0 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c580035f0 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5800ce60 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.477+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6819fe50 con 0x7f8c681005a0 2024-03-28T15:52:19.483 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.478+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c681078e0 con 0x7f8c681005a0 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.479+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8c580037a0 con 0x7f8c681005a0 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.480+0000 7f8c657fa640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.480+0000 7f8c677fe640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.480+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8c580cf100 con 0x7f8c681005a0 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.480+0000 7f8c677fe640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 0x7f8c500a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02c7b0 tx=0x7f8c5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.484 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.483+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8c5809be40 con 0x7f8c681005a0 2024-03-28T15:52:19.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.665+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8c6802d220 con 0x7f8c681005a0 2024-03-28T15:52:19.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.665+0000 7f8c657fa640 1 -- 172.21.15.23:0/316985536 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8c5809b740 con 0x7f8c681005a0 2024-03-28T15:52:19.667 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 msgr2=0x7f8c500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 0x7f8c500a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8c5c02c7b0 tx=0x7f8c5c005fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 msgr2=0x7f8c68069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8c5800e9c0 tx=0x7f8c5800ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 shutdown_connections 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f8c500a2800 0x7f8c500a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8c68102150 0x7f8c6819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8c68101790 0x7f8c6819c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.667+0000 7f8c6e39d640 1 --2- 172.21.15.23:0/316985536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8c681005a0 0x7f8c68069370 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.668+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 >> 172.21.15.23:0/316985536 conn(0x7f8c680fbb30 msgr2=0x7f8c68104ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:19.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.668+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 shutdown_connections 2024-03-28T15:52:19.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.668+0000 7f8c6e39d640 1 -- 172.21.15.23:0/316985536 wait complete. 2024-03-28T15:52:19.680 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 -- 172.21.15.23:0/3671672392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 msgr2=0x7f2a78104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/3671672392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a78104530 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c009900 tx=0x7f2a6c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 -- 172.21.15.23:0/3671672392 shutdown_connections 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/3671672392 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a781073a0 0x7f2a781097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/3671672392 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a78104a70 0x7f2a78106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/3671672392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a78104530 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.789 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 -- 172.21.15.23:0/3671672392 >> 172.21.15.23:0/3671672392 conn(0x7f2a780fbb50 msgr2=0x7f2a780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:19.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.787+0000 7f2a7eedc640 1 -- 172.21.15.23:0/3671672392 shutdown_connections 2024-03-28T15:52:19.790 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 -- 172.21.15.23:0/3671672392 wait complete. 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 Processor -- start 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 -- start start 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 0x7f2a78197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a781073a0 0x7f2a7819efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7810d350 con 0x7f2a78104a70 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7810d1d0 con 0x7f2a78102140 2024-03-28T15:52:19.792 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.788+0000 7f2a7eedc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7810d4d0 con 0x7f2a781073a0 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 0x7f2a78197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7cc78640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7cc78640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:54836/0 (socket says 172.21.15.23:54836) 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7cc78640 1 -- 172.21.15.23:0/2379265095 learned_addr learned my addr 172.21.15.23:0/2379265095 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7d479640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a781073a0 0x7f2a7819efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 -- 172.21.15.23:0/2379265095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a781073a0 msgr2=0x7f2a7819efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a781073a0 0x7f2a7819efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 -- 172.21.15.23:0/2379265095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 msgr2=0x7f2a781979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 -- 172.21.15.23:0/2379265095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7819f610 con 0x7f2a78104a70 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7cc78640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a77fff640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 0x7f2a78197f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400e9c0 tx=0x7f2a6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.793 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6400ccc0 con 0x7f2a78104a70 2024-03-28T15:52:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7819f900 con 0x7f2a78104a70 2024-03-28T15:52:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a64010910 con 0x7f2a78104a70 2024-03-28T15:52:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64018880 con 0x7f2a78104a70 2024-03-28T15:52:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.789+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7819fea0 con 0x7f2a78104a70 2024-03-28T15:52:19.794 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.791+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a780fd640 con 0x7f2a78104a70 2024-03-28T15:52:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.792+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2a64010ac0 con 0x7f2a78104a70 2024-03-28T15:52:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.792+0000 7f2a75ffb640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 0x7f2a680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.792+0000 7f2a7cc78640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 0x7f2a680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.793+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2a640cf120 con 0x7f2a78104a70 2024-03-28T15:52:19.795 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.793+0000 7f2a7cc78640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 0x7f2a680a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c000c00 tx=0x7f2a6c02c880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:19.798 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:19.797+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a6409b810 con 0x7f2a78104a70 2024-03-28T15:52:20.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.000+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a780622a0 con 0x7f2a78104a70 2024-03-28T15:52:20.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.000+0000 7f2a75ffb640 1 -- 172.21.15.23:0/2379265095 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2a6409f8b0 con 0x7f2a78104a70 2024-03-28T15:52:20.001 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:20.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 msgr2=0x7f2a680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 0x7f2a680a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c000c00 tx=0x7f2a6c02c880 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 msgr2=0x7f2a78197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 0x7f2a78197f30 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400e9c0 tx=0x7f2a6400ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 shutdown_connections 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f2a680a2800 0x7f2a680a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f2a781073a0 0x7f2a7819efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2a78104a70 0x7f2a78197f30 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 --2- 172.21.15.23:0/2379265095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a78102140 0x7f2a781979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.002+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 >> 172.21.15.23:0/2379265095 conn(0x7f2a780fbb50 msgr2=0x7f2a78102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.003+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 shutdown_connections 2024-03-28T15:52:20.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.003+0000 7f2a7eedc640 1 -- 172.21.15.23:0/2379265095 wait complete. 2024-03-28T15:52:20.015 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.120+0000 7fcd20dc6640 1 -- 172.21.15.23:0/3717571761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c1005a0 msgr2=0x7fcd1c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.120+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/3717571761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c1009c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcd10009900 tx=0x7fcd1002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 -- 172.21.15.23:0/3717571761 shutdown_connections 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/3717571761 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 0x7fcd1c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/3717571761 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c101790 0x7fcd1c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/3717571761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c1009c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 -- 172.21.15.23:0/3717571761 >> 172.21.15.23:0/3717571761 conn(0x7fcd1c0fbb30 msgr2=0x7fcd1c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 -- 172.21.15.23:0/3717571761 shutdown_connections 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 -- 172.21.15.23:0/3717571761 wait complete. 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 Processor -- start 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 -- start start 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c101790 0x7fcd1c19c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.121+0000 7fcd20dc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 0x7fcd1c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd20dc6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c10c230 con 0x7fcd1c1005a0 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd20dc6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c10c0b0 con 0x7fcd1c101790 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd20dc6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c10c3b0 con 0x7fcd1c102150 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58024/0 (socket says 172.21.15.23:58024) 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 -- 172.21.15.23:0/4266504757 learned_addr learned my addr 172.21.15.23:0/4266504757 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd19d9b640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c101790 0x7fcd1c19c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1ad9d640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 0x7fcd1c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 -- 172.21.15.23:0/4266504757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 msgr2=0x7fcd1c19ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 0x7fcd1c19ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 -- 172.21.15.23:0/4266504757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c101790 msgr2=0x7fcd1c19c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c101790 0x7fcd1c19c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 -- 172.21.15.23:0/4266504757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c19f640 con 0x7fcd1c1005a0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd1a59c640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd10000c00 tx=0x7fcd1002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd10041070 con 0x7fcd1c1005a0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c19f8d0 con 0x7fcd1c1005a0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd10004450 con 0x7fcd1c1005a0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd10034440 con 0x7fcd1c1005a0 2024-03-28T15:52:20.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.122+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c19fdc0 con 0x7fcd1c1005a0 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.123+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c107990 con 0x7fcd1c1005a0 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.125+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd10044070 con 0x7fcd1c1005a0 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.125+0000 7fcd0b7fe640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 0x7fcd040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.125+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcd100f0a90 con 0x7fcd1c1005a0 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.125+0000 7fcd19d9b640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 0x7fcd040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.126+0000 7fcd19d9b640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 0x7fcd040a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c0095f0 tx=0x7fcd0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.129+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcd100bd210 con 0x7fcd1c1005a0 2024-03-28T15:52:20.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:19 smithi117 ceph-mon[36002]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T15:52:20.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:19 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1284067212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:20.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:19 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/316985536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:20.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.322+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd1c02d200 con 0x7fcd1c1005a0 2024-03-28T15:52:20.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.322+0000 7fcd0b7fe640 1 -- 172.21.15.23:0/4266504757 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd100c12b0 con 0x7fcd1c1005a0 2024-03-28T15:52:20.323 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:52:20.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 msgr2=0x7fcd040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 0x7fcd040a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c0095f0 tx=0x7fcd0c005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:20.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 msgr2=0x7fcd1c069390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcd10000c00 tx=0x7fcd1002c640 comp rx=0 tx=0).stop 2024-03-28T15:52:20.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 shutdown_connections 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcd040a2800 0x7fcd040a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd1c102150 0x7fcd1c19ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd1c101790 0x7fcd1c19c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 --2- 172.21.15.23:0/4266504757 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd1c1005a0 0x7fcd1c069390 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 >> 172.21.15.23:0/4266504757 conn(0x7fcd1c0fbb30 msgr2=0x7fcd1c104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 shutdown_connections 2024-03-28T15:52:20.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.324+0000 7fcd20dc6640 1 -- 172.21.15.23:0/4266504757 wait complete. 2024-03-28T15:52:20.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:19 smithi023 ceph-mon[32448]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T15:52:20.335 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:19 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1284067212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:20.336 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:19 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/316985536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:20.337 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:52:20.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:19 smithi120 ceph-mon[36826]: pgmap v344: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-28T15:52:20.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:19 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1284067212' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:20.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:19 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/316985536' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 -- 172.21.15.23:0/2132844187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c104a70 msgr2=0x7f1d7c106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 --2- 172.21.15.23:0/2132844187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c104a70 0x7f1d7c106e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1d70009860 tx=0x7f1d7002bb70 comp rx=0 tx=0).stop 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 -- 172.21.15.23:0/2132844187 shutdown_connections 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 --2- 172.21.15.23:0/2132844187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c1073a0 0x7f1d7c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 --2- 172.21.15.23:0/2132844187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c104a70 0x7f1d7c106e60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 --2- 172.21.15.23:0/2132844187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 -- 172.21.15.23:0/2132844187 >> 172.21.15.23:0/2132844187 conn(0x7f1d7c0fbb50 msgr2=0x7f1d7c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 -- 172.21.15.23:0/2132844187 shutdown_connections 2024-03-28T15:52:20.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.441+0000 7f1d82310640 1 -- 172.21.15.23:0/2132844187 wait complete. 2024-03-28T15:52:20.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 Processor -- start 2024-03-28T15:52:20.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 -- start start 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c1073a0 0x7f1d7c19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10d350 con 0x7f1d7c104a70 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10d1d0 con 0x7f1d7c102140 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d82310640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c10d4d0 con 0x7f1d7c1073a0 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d7b7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58044/0 (socket says 172.21.15.23:58044) 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d7b7fe640 1 -- 172.21.15.23:0/1666144241 learned_addr learned my addr 172.21.15.23:0/1666144241 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d7bfff640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.442+0000 7f1d808ad640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c1073a0 0x7f1d7c19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 -- 172.21.15.23:0/1666144241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c1073a0 msgr2=0x7f1d7c19efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c1073a0 0x7f1d7c19efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 -- 172.21.15.23:0/1666144241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 msgr2=0x7f1d7c197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 -- 172.21.15.23:0/1666144241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c19f620 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7bfff640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d7b7fe640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1d700349e0 tx=0x7f1d70032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70041020 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d7002c9f0 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d7c19f8b0 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d70039040 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.443+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c19fda0 con 0x7f1d7c104a70 2024-03-28T15:52:20.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.444+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d7c0fd640 con 0x7f1d7c104a70 2024-03-28T15:52:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.445+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1d70044020 con 0x7f1d7c104a70 2024-03-28T15:52:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.446+0000 7f1d797fa640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 0x7f1d640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.446+0000 7f1d7bfff640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 0x7f1d640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.446+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f1d700f0620 con 0x7f1d7c104a70 2024-03-28T15:52:20.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.446+0000 7f1d7bfff640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 0x7f1d640a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009fc0 tx=0x7f1d6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.450+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1d7004b020 con 0x7f1d7c104a70 2024-03-28T15:52:20.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.636+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1d7c0622a0 con 0x7f1d7c104a70 2024-03-28T15:52:20.638 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.637+0000 7f1d797fa640 1 -- 172.21.15.23:0/1666144241 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1d700bd360 con 0x7f1d7c104a70 2024-03-28T15:52:20.638 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:20.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 msgr2=0x7f1d640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 0x7f1d640a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009fc0 tx=0x7f1d6c005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:20.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 msgr2=0x7f1d7c197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1d700349e0 tx=0x7f1d70032430 comp rx=0 tx=0).stop 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 shutdown_connections 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f1d640a2800 0x7f1d640a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1d7c1073a0 0x7f1d7c19efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1d7c104a70 0x7f1d7c197f40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 --2- 172.21.15.23:0/1666144241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d7c102140 0x7f1d7c197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 >> 172.21.15.23:0/1666144241 conn(0x7f1d7c0fbb50 msgr2=0x7f1d7c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 shutdown_connections 2024-03-28T15:52:20.641 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.639+0000 7f1d82310640 1 -- 172.21.15.23:0/1666144241 wait complete. 2024-03-28T15:52:20.651 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:52:20.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 -- 172.21.15.23:0/447213519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4534104ae0 msgr2=0x7f4534106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 --2- 172.21.15.23:0/447213519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4534104ae0 0x7f4534106ed0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4528009900 tx=0x7f452802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:20.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 -- 172.21.15.23:0/447213519 shutdown_connections 2024-03-28T15:52:20.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 --2- 172.21.15.23:0/447213519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534107410 0x7f4534109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 --2- 172.21.15.23:0/447213519 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4534104ae0 0x7f4534106ed0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 --2- 172.21.15.23:0/447213519 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f45341021b0 0x7f45341045a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 -- 172.21.15.23:0/447213519 >> 172.21.15.23:0/447213519 conn(0x7f45340fbb80 msgr2=0x7f45340fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 -- 172.21.15.23:0/447213519 shutdown_connections 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 -- 172.21.15.23:0/447213519 wait complete. 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.753+0000 7f45397cd640 1 Processor -- start 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 -- start start 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534104ae0 0x7f4534197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4534107410 0x7f453419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453410d320 con 0x7f45341021b0 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453410d1a0 con 0x7f4534104ae0 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45397cd640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453410d4a0 con 0x7f4534107410 2024-03-28T15:52:20.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58066/0 (socket says 172.21.15.23:58066) 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 -- 172.21.15.23:0/3260390024 learned_addr learned my addr 172.21.15.23:0/3260390024 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45327fc640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534104ae0 0x7f4534197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f45337fe640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4534107410 0x7f453419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 -- 172.21.15.23:0/3260390024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4534107410 msgr2=0x7f453419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4534107410 0x7f453419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 -- 172.21.15.23:0/3260390024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534104ae0 msgr2=0x7f4534197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534104ae0 0x7f4534197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 -- 172.21.15.23:0/3260390024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453419f640 con 0x7f45341021b0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f4532ffd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f452400e9f0 tx=0x7f452400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.754+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f452400ccc0 con 0x7f45341021b0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.755+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453419f930 con 0x7f45341021b0 2024-03-28T15:52:20.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.755+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4524010910 con 0x7f45341021b0 2024-03-28T15:52:20.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.755+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45240188a0 con 0x7f45341021b0 2024-03-28T15:52:20.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.755+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453419fea0 con 0x7f45341021b0 2024-03-28T15:52:20.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.756+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45340688e0 con 0x7f45341021b0 2024-03-28T15:52:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.757+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4524010ac0 con 0x7f45341021b0 2024-03-28T15:52:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.757+0000 7f451bfff640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 0x7f451c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.757+0000 7f45327fc640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 0x7f451c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.757+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f45240cf200 con 0x7f45341021b0 2024-03-28T15:52:20.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.758+0000 7f45327fc640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 0x7f451c0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4528000c00 tx=0x7f4528005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:20.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.762+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f452409bf40 con 0x7f45341021b0 2024-03-28T15:52:20.984 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.983+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45340fd670 con 0x7f45341021b0 2024-03-28T15:52:20.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.984+0000 7f451bfff640 1 -- 172.21.15.23:0/3260390024 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f452409b840 con 0x7f45341021b0 2024-03-28T15:52:20.985 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:20.985 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:52:20.986 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:20.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 msgr2=0x7f451c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 0x7f451c0a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4528000c00 tx=0x7f4528005fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.987 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 msgr2=0x7f4534197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f452400e9f0 tx=0x7f452400edf0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 shutdown_connections 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f451c0a2800 0x7f451c0a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4534107410 0x7f453419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4534104ae0 0x7f4534197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 --2- 172.21.15.23:0/3260390024 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f45341021b0 0x7f4534197a20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 >> 172.21.15.23:0/3260390024 conn(0x7f45340fbb80 msgr2=0x7f4534102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 shutdown_connections 2024-03-28T15:52:20.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:20.986+0000 7f45397cd640 1 -- 172.21.15.23:0/3260390024 wait complete. 2024-03-28T15:52:20.999 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/1082195887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd081021f0 msgr2=0x7fcd08108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/1082195887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd081021f0 0x7fcd08108a80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fccf8009ca0 tx=0x7fccf802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/1082195887 shutdown_connections 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/1082195887 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd081021f0 0x7fcd08108a80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/1082195887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08101830 0x7fcd08101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/1082195887 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd08100640 0x7fcd08100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/1082195887 >> 172.21.15.23:0/1082195887 conn(0x7fcd080fbbd0 msgr2=0x7fcd080fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/1082195887 shutdown_connections 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.101+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/1082195887 wait complete. 2024-03-28T15:52:21.103 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 Processor -- start 2024-03-28T15:52:21.104 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 -- start start 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 0x7fcd08196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd081021f0 0x7fcd0819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0810c250 con 0x7fcd08101830 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0810c0d0 con 0x7fcd08100640 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd0d8c6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0810c3d0 con 0x7fcd081021f0 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd067fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd067fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58082/0 (socket says 172.21.15.23:58082) 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd06ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 0x7fcd08196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd067fc640 1 -- 172.21.15.23:0/3308617186 learned_addr learned my addr 172.21.15.23:0/3308617186 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.102+0000 7fcd077fe640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd081021f0 0x7fcd0819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 -- 172.21.15.23:0/3308617186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd081021f0 msgr2=0x7fcd0819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.106 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd081021f0 0x7fcd0819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 -- 172.21.15.23:0/3308617186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 msgr2=0x7fcd08196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 0x7fcd08196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 -- 172.21.15.23:0/3308617186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0819e490 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd06ffd640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 0x7fcd08196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd067fc640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fccf000ea60 tx=0x7fccf000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000cde0 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf00035f0 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0819e780 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0010670 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.103+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0819ecd0 con 0x7fcd08101830 2024-03-28T15:52:21.107 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.104+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd08107960 con 0x7fcd08101830 2024-03-28T15:52:21.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.106+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fccf00037a0 con 0x7fcd08101830 2024-03-28T15:52:21.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.106+0000 7fcd0c8c4640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 0x7fccf40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.107+0000 7fcd06ffd640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 0x7fccf40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.107+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fccf00cf1d0 con 0x7fcd08101830 2024-03-28T15:52:21.108 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.107+0000 7fcd06ffd640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 0x7fccf40a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd08101690 tx=0x7fcd00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.111 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.110+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fccf009bf40 con 0x7fcd08101830 2024-03-28T15:52:21.295 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.294+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0802d240 con 0x7fcd08101830 2024-03-28T15:52:21.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.295+0000 7fcd0c8c4640 1 -- 172.21.15.23:0/3308617186 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24622 (secure 0 0 0) 0x7fccf009b840 con 0x7fcd08101830 2024-03-28T15:52:21.297 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:21.297 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:51:35.345652+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 msgr2=0x7fccf40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 0x7fccf40a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd08101690 tx=0x7fcd00005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 msgr2=0x7fcd08196db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fccf000ea60 tx=0x7fccf000ef20 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 shutdown_connections 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fccf40a2800 0x7fccf40a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcd081021f0 0x7fcd0819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcd08101830 0x7fcd08196db0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 --2- 172.21.15.23:0/3308617186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd08100640 0x7fcd08196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 >> 172.21.15.23:0/3308617186 conn(0x7fcd080fbbd0 msgr2=0x7fcd08104b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.297+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 shutdown_connections 2024-03-28T15:52:21.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.298+0000 7fcd0d8c6640 1 -- 172.21.15.23:0/3308617186 wait complete. 2024-03-28T15:52:21.312 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:20 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2379265095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.312 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:20 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4266504757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.313 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:20 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1666144241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.314 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:52:21.314 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:52:21.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:20 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2379265095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:20 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4266504757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:20 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1666144241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:20 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2379265095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:20 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4266504757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:20 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1666144241' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:21.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 -- 172.21.15.23:0/59266999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac1016d0 msgr2=0x7febac101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 --2- 172.21.15.23:0/59266999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac1016d0 0x7febac101af0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009900 tx=0x7feb9c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 -- 172.21.15.23:0/59266999 shutdown_connections 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 --2- 172.21.15.23:0/59266999 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac103280 0x7febac109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 --2- 172.21.15.23:0/59266999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1028c0 0x7febac102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 --2- 172.21.15.23:0/59266999 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac1016d0 0x7febac101af0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 -- 172.21.15.23:0/59266999 >> 172.21.15.23:0/59266999 conn(0x7febac0fcc40 msgr2=0x7febac0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 -- 172.21.15.23:0/59266999 shutdown_connections 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.419+0000 7febb0e7e640 1 -- 172.21.15.23:0/59266999 wait complete. 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 Processor -- start 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 -- start start 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1016d0 0x7febac197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 0x7febac197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac10d320 con 0x7febac103280 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac10d1a0 con 0x7febac1016d0 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febb0e7e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac10d4a0 con 0x7febac1028c0 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58096/0 (socket says 172.21.15.23:58096) 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7feba9d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 0x7febac197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 -- 172.21.15.23:0/2259278490 learned_addr learned my addr 172.21.15.23:0/2259278490 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaa59c640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1016d0 0x7febac197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 -- 172.21.15.23:0/2259278490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 msgr2=0x7febac197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 0x7febac197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 -- 172.21.15.23:0/2259278490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1016d0 msgr2=0x7febac197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1016d0 0x7febac197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7febaad9d640 1 -- 172.21.15.23:0/2259278490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac19f620 con 0x7febac103280 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.420+0000 7feba9d9b640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 0x7febac197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7febaad9d640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feba400eb10 tx=0x7feba400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4004b20 con 0x7febac103280 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac19f910 con 0x7febac103280 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feba4004cc0 con 0x7febac103280 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4010760 con 0x7febac103280 2024-03-28T15:52:21.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.421+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac19fe60 con 0x7febac103280 2024-03-28T15:52:21.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.422+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac108a10 con 0x7febac103280 2024-03-28T15:52:21.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.423+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feba4005000 con 0x7febac103280 2024-03-28T15:52:21.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.424+0000 7feb9b7fe640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 0x7feb940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.424+0000 7febaa59c640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 0x7feb940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.424+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7feba40cf560 con 0x7febac103280 2024-03-28T15:52:21.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.424+0000 7febaa59c640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 0x7feb940a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feb9c000c00 tx=0x7feb9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.428+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feba4053740 con 0x7febac103280 2024-03-28T15:52:21.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.614+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7febac02d2c0 con 0x7febac103280 2024-03-28T15:52:21.615 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.614+0000 7feb9b7fe640 1 -- 172.21.15.23:0/2259278490 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7feba409c2a0 con 0x7febac103280 2024-03-28T15:52:21.615 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:52:21.617 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 msgr2=0x7feb940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 0x7feb940a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feb9c000c00 tx=0x7feb9c005fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 msgr2=0x7febac19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feba400eb10 tx=0x7feba400c2d0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 shutdown_connections 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7feb940a2800 0x7feb940a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7febac103280 0x7febac19efb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7febac1028c0 0x7febac197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 --2- 172.21.15.23:0/2259278490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febac1016d0 0x7febac197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 >> 172.21.15.23:0/2259278490 conn(0x7febac0fcc40 msgr2=0x7febac105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.618 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 shutdown_connections 2024-03-28T15:52:21.619 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.616+0000 7febb0e7e640 1 -- 172.21.15.23:0/2259278490 wait complete. 2024-03-28T15:52:21.629 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:52:21.737 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 -- 172.21.15.23:0/2727009549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc1005a0 msgr2=0x7f06bc1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 --2- 172.21.15.23:0/2727009549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc1005a0 0x7f06bc1009c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009900 tx=0x7f06ac02bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 -- 172.21.15.23:0/2727009549 shutdown_connections 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 --2- 172.21.15.23:0/2727009549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 0x7f06bc1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 --2- 172.21.15.23:0/2727009549 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc101790 0x7f06bc101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 --2- 172.21.15.23:0/2727009549 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc1005a0 0x7f06bc1009c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 -- 172.21.15.23:0/2727009549 >> 172.21.15.23:0/2727009549 conn(0x7f06bc0fbb30 msgr2=0x7f06bc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 -- 172.21.15.23:0/2727009549 shutdown_connections 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.734+0000 7f06c080b640 1 -- 172.21.15.23:0/2727009549 wait complete. 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 Processor -- start 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 -- start start 2024-03-28T15:52:21.738 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc101790 0x7f06bc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 0x7f06bc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc10c230 con 0x7f06bc1005a0 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc10c0b0 con 0x7f06bc102150 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06c080b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc10c3b0 con 0x7f06bc101790 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06ba59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06ba59c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58112/0 (socket says 172.21.15.23:58112) 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06ba59c640 1 -- 172.21.15.23:0/789082816 learned_addr learned my addr 172.21.15.23:0/789082816 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06bad9d640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 0x7f06bc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.735+0000 7f06b9d9b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc101790 0x7f06bc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 -- 172.21.15.23:0/789082816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc101790 msgr2=0x7f06bc19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc101790 0x7f06bc19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 -- 172.21.15.23:0/789082816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 msgr2=0x7f06bc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.739 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 0x7f06bc1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 -- 172.21.15.23:0/789082816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc1a1800 con 0x7f06bc1005a0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ba59c640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f06ac000c00 tx=0x7f06ac02c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac041070 con 0x7f06bc1005a0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06ac004450 con 0x7f06bc1005a0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06bc1a1a90 con 0x7f06bc1005a0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac034440 con 0x7f06bc1005a0 2024-03-28T15:52:21.740 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.736+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc1a1f80 con 0x7f06bc1005a0 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.737+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc1a2150 con 0x7f06bc1005a0 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.738+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f06ac044070 con 0x7f06bc1005a0 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.739+0000 7f06ab7fe640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 0x7f06a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.739+0000 7f06b9d9b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 0x7f06a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.739+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f06ac0f0a90 con 0x7f06bc1005a0 2024-03-28T15:52:21.741 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.739+0000 7f06b9d9b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 0x7f06a40a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f06bc19b0f0 tx=0x7f06b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:21.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.743+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06ac0bd210 con 0x7f06bc1005a0 2024-03-28T15:52:21.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.967+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f06bc1a24e0 con 0x7f06bc1005a0 2024-03-28T15:52:21.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.969+0000 7f06ab7fe640 1 -- 172.21.15.23:0/789082816 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f06ac0c12b0 con 0x7f06bc1005a0 2024-03-28T15:52:21.970 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 msgr2=0x7f06a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 0x7f06a40a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f06bc19b0f0 tx=0x7f06b0005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 msgr2=0x7f06bc199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f06ac000c00 tx=0x7f06ac02c640 comp rx=0 tx=0).stop 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 shutdown_connections 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f06a40a2800 0x7f06a40a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06bc102150 0x7f06bc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.972 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f06bc101790 0x7f06bc19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 --2- 172.21.15.23:0/789082816 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f06bc1005a0 0x7f06bc199be0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:21.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 >> 172.21.15.23:0/789082816 conn(0x7f06bc0fbb30 msgr2=0x7f06bc0fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:21.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 shutdown_connections 2024-03-28T15:52:21.973 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:21.971+0000 7f06c080b640 1 -- 172.21.15.23:0/789082816 wait complete. 2024-03-28T15:52:21.983 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.108+0000 7f5ffb885640 1 -- 172.21.15.23:0/2738974367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4104a50 msgr2=0x7f5ff4106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.108+0000 7f5ffb885640 1 --2- 172.21.15.23:0/2738974367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4104a50 0x7f5ff4106e40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009900 tx=0x7f5fe402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 -- 172.21.15.23:0/2738974367 shutdown_connections 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 --2- 172.21.15.23:0/2738974367 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4107380 0x7f5ff41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 --2- 172.21.15.23:0/2738974367 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4104a50 0x7f5ff4106e40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 --2- 172.21.15.23:0/2738974367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 0x7f5ff4104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.112 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 -- 172.21.15.23:0/2738974367 >> 172.21.15.23:0/2738974367 conn(0x7f5ff40fbb30 msgr2=0x7f5ff40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 -- 172.21.15.23:0/2738974367 shutdown_connections 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 -- 172.21.15.23:0/2738974367 wait complete. 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 Processor -- start 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.109+0000 7f5ffb885640 1 -- start start 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 0x7f5ff4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4104a50 0x7f5ff4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff410d300 con 0x7f5ff4107380 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff410d180 con 0x7f5ff4102120 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff410d480 con 0x7f5ff4104a50 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58130/0 (socket says 172.21.15.23:58130) 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 -- 172.21.15.23:0/3756622442 learned_addr learned my addr 172.21.15.23:0/3756622442 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:22.113 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9621640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 0x7f5ff4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff8e20640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4104a50 0x7f5ff4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 -- 172.21.15.23:0/3756622442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4104a50 msgr2=0x7f5ff4197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4104a50 0x7f5ff4197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 -- 172.21.15.23:0/3756622442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 msgr2=0x7f5ff4197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 0x7f5ff4197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 -- 172.21.15.23:0/3756622442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff419f620 con 0x7f5ff4107380 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ff9e22640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00eb10 tx=0x7f5fec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec004b20 con 0x7f5ff4107380 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fec004cc0 con 0x7f5ff4107380 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff419f910 con 0x7f5ff4107380 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.110+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec010760 con 0x7f5ff4107380 2024-03-28T15:52:22.114 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.111+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff419fe30 con 0x7f5ff4107380 2024-03-28T15:52:22.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.112+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff40fd620 con 0x7f5ff4107380 2024-03-28T15:52:22.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.113+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5fec005000 con 0x7f5ff4107380 2024-03-28T15:52:22.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.114+0000 7f5fea7fc640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 0x7f5fe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.114+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f5fec0cf4e0 con 0x7f5ff4107380 2024-03-28T15:52:22.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.114+0000 7f5ff9621640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 0x7f5fe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.116 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.114+0000 7f5ff9621640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 0x7f5fe00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00095f0 tx=0x7f5ff0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.119 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.118+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5fec053740 con 0x7f5ff4107380 2024-03-28T15:52:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.301+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ff402d070 con 0x7f5ff4107380 2024-03-28T15:52:22.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.301+0000 7f5fea7fc640 1 -- 172.21.15.23:0/3756622442 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5fec09c220 con 0x7f5ff4107380 2024-03-28T15:52:22.303 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:52:22.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 msgr2=0x7f5fe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 0x7f5fe00a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5ff00095f0 tx=0x7f5ff0005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 msgr2=0x7f5ff419efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5fec00eb10 tx=0x7f5fec00c2d0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 shutdown_connections 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f5fe00a2880 0x7f5fe00a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5ff4107380 0x7f5ff419efb0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5ff4104a50 0x7f5ff4197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 --2- 172.21.15.23:0/3756622442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ff4102120 0x7f5ff4197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 >> 172.21.15.23:0/3756622442 conn(0x7f5ff40fbb30 msgr2=0x7f5ff4102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 shutdown_connections 2024-03-28T15:52:22.305 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.303+0000 7f5ffb885640 1 -- 172.21.15.23:0/3756622442 wait complete. 2024-03-28T15:52:22.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:21 smithi023 ceph-mon[32448]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T15:52:22.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:21 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3260390024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:21 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3308617186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.318 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:21 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2259278490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:22.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:52:22.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:21 smithi117 ceph-mon[36002]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T15:52:22.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:21 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3260390024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:21 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3308617186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:21 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2259278490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:21 smithi120 ceph-mon[36826]: pgmap v345: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-28T15:52:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:21 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3260390024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:21 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3308617186' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:22.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:21 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2259278490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:22.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 -- 172.21.15.23:0/960082528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8107380 msgr2=0x7effa81097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 --2- 172.21.15.23:0/960082528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8107380 0x7effa81097d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7effa0009ca0 tx=0x7effa002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 -- 172.21.15.23:0/960082528 shutdown_connections 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 --2- 172.21.15.23:0/960082528 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8107380 0x7effa81097d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 --2- 172.21.15.23:0/960082528 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8104a50 0x7effa8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 --2- 172.21.15.23:0/960082528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa8104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 -- 172.21.15.23:0/960082528 >> 172.21.15.23:0/960082528 conn(0x7effa80fbb30 msgr2=0x7effa80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 -- 172.21.15.23:0/960082528 shutdown_connections 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.430+0000 7effae90c640 1 -- 172.21.15.23:0/960082528 wait complete. 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 Processor -- start 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 -- start start 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8107380 0x7effa819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d320 con 0x7effa8104a50 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d1a0 con 0x7effa8102120 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effae90c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810d4a0 con 0x7effa8107380 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa77fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58156/0 (socket says 172.21.15.23:58156) 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa77fe640 1 -- 172.21.15.23:0/2368275269 learned_addr learned my addr 172.21.15.23:0/2368275269 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa7fff640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effacea9640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8107380 0x7effa819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa77fe640 1 -- 172.21.15.23:0/2368275269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8107380 msgr2=0x7effa819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.431+0000 7effa77fe640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8107380 0x7effa819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa77fe640 1 -- 172.21.15.23:0/2368275269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 msgr2=0x7effa81979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa77fe640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa81979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa77fe640 1 -- 172.21.15.23:0/2368275269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa819f5f0 con 0x7effa8104a50 2024-03-28T15:52:22.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa7fff640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa77fe640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e9c0 tx=0x7eff9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c00cd80 con 0x7effa8104a50 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff9c005500 con 0x7effa8104a50 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa819f8e0 con 0x7effa8104a50 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff9c010690 con 0x7effa8104a50 2024-03-28T15:52:22.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.432+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa819fe30 con 0x7effa8104a50 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.433+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa80fd620 con 0x7effa8104a50 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.434+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7eff9c0046c0 con 0x7effa8104a50 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.435+0000 7effa57fa640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.435+0000 7effa7fff640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.435+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7eff9c0cf120 con 0x7effa8104a50 2024-03-28T15:52:22.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.435+0000 7effa7fff640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff98009fc0 tx=0x7eff98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.439+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff9c09be60 con 0x7effa8104a50 2024-03-28T15:52:22.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.621+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7effa80622a0 con 0x7effa8104a50 2024-03-28T15:52:22.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.621+0000 7effa57fa640 1 -- 172.21.15.23:0/2368275269 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7eff9c09b760 con 0x7effa8104a50 2024-03-28T15:52:22.623 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 msgr2=0x7eff900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff98009fc0 tx=0x7eff98005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 msgr2=0x7effa8197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7eff9c00e9c0 tx=0x7eff9c00ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 shutdown_connections 2024-03-28T15:52:22.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7eff900a2800 0x7eff900a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effa8107380 0x7effa819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effa8104a50 0x7effa8197f10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 --2- 172.21.15.23:0/2368275269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effa8102120 0x7effa81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 >> 172.21.15.23:0/2368275269 conn(0x7effa80fbb30 msgr2=0x7effa81029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 shutdown_connections 2024-03-28T15:52:22.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.624+0000 7effae90c640 1 -- 172.21.15.23:0/2368275269 wait complete. 2024-03-28T15:52:22.637 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:52:22.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 -- 172.21.15.23:0/3268037702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb068102140 msgr2=0x7fb068104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 --2- 172.21.15.23:0/3268037702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb068102140 0x7fb068104530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb058009860 tx=0x7fb05802bb70 comp rx=0 tx=0).stop 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 -- 172.21.15.23:0/3268037702 shutdown_connections 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 --2- 172.21.15.23:0/3268037702 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0681073a0 0x7fb0681097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 --2- 172.21.15.23:0/3268037702 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068104a70 0x7fb068106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 --2- 172.21.15.23:0/3268037702 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb068102140 0x7fb068104530 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 -- 172.21.15.23:0/3268037702 >> 172.21.15.23:0/3268037702 conn(0x7fb0680fbb30 msgr2=0x7fb0680fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 -- 172.21.15.23:0/3268037702 shutdown_connections 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 -- 172.21.15.23:0/3268037702 wait complete. 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.739+0000 7fb06db53640 1 Processor -- start 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 -- start start 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068104a70 0x7fb0681000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0681073a0 0x7fb068100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06810d370 con 0x7fb068102140 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06810d1f0 con 0x7fb0681073a0 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb06db53640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06810d4f0 con 0x7fb068104a70 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58184/0 (socket says 172.21.15.23:58184) 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 -- 172.21.15.23:0/1022178021 learned_addr learned my addr 172.21.15.23:0/1022178021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb066ffd640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068104a70 0x7fb0681000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb067fff640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0681073a0 0x7fb068100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 -- 172.21.15.23:0/1022178021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068104a70 msgr2=0x7fb0681000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068104a70 0x7fb0681000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 -- 172.21.15.23:0/1022178021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0681073a0 msgr2=0x7fb068100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0681073a0 0x7fb068100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 -- 172.21.15.23:0/1022178021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb068100ec0 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.740+0000 7fb0677fe640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb058005940 tx=0x7fb058032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.741+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058041020 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.741+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0681a1aa0 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.741+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb05802c9f0 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.741+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058039040 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.741+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0681a1ff0 con 0x7fb068102140 2024-03-28T15:52:22.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.742+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0680fd620 con 0x7fb068102140 2024-03-28T15:52:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.743+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb058044020 con 0x7fb068102140 2024-03-28T15:52:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.744+0000 7fb064ff9640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 0x7fb0500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.744+0000 7fb066ffd640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 0x7fb0500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.744+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb0580f0710 con 0x7fb068102140 2024-03-28T15:52:22.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.744+0000 7fb066ffd640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 0x7fb0500a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb068101820 tx=0x7fb05c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:22.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.748+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb05804b020 con 0x7fb068102140 2024-03-28T15:52:22.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.969+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06802d070 con 0x7fb068102140 2024-03-28T15:52:22.971 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.970+0000 7fb064ff9640 1 -- 172.21.15.23:0/1022178021 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7fb0580bd450 con 0x7fb068102140 2024-03-28T15:52:22.972 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:22.972 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:52:22.972 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 msgr2=0x7fb0500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 0x7fb0500a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb068101820 tx=0x7fb05c009260 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 msgr2=0x7fb068101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb058005940 tx=0x7fb058032470 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 shutdown_connections 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0500a2800 0x7fb0500a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0681073a0 0x7fb068100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb068104a70 0x7fb0681000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 --2- 172.21.15.23:0/1022178021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb068102140 0x7fb068101ab0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.972+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 >> 172.21.15.23:0/1022178021 conn(0x7fb0680fbb30 msgr2=0x7fb0681029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.973+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 shutdown_connections 2024-03-28T15:52:22.974 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:22.973+0000 7fb06db53640 1 -- 172.21.15.23:0/1022178021 wait complete. 2024-03-28T15:52:22.985 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:23.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/632157299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 msgr2=0x7f9c44109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.090 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/632157299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c44109b00 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c009ca0 tx=0x7f9c3c02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/632157299 shutdown_connections 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/632157299 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c44109b00 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/632157299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 0x7f9c44102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/632157299 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 0x7f9c44101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/632157299 >> 172.21.15.23:0/632157299 conn(0x7f9c440fcc50 msgr2=0x7f9c440ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/632157299 shutdown_connections 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.086+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/632157299 wait complete. 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 Processor -- start 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 -- start start 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 0x7f9c44197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 0x7f9c44197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410d340 con 0x7f9c44103270 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410d1c0 con 0x7f9c441028b0 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c4a6bf640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4410d4c0 con 0x7f9c441016c0 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58208/0 (socket says 172.21.15.23:58208) 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 -- 172.21.15.23:0/3840536255 learned_addr learned my addr 172.21.15.23:0/3840536255 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c437fe640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 0x7f9c44197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c43fff640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 0x7f9c44197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.092 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 -- 172.21.15.23:0/3840536255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 msgr2=0x7f9c44197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 0x7f9c44197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 -- 172.21.15.23:0/3840536255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 msgr2=0x7f9c44197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.087+0000 7f9c48c5c640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 0x7f9c44197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c48c5c640 1 -- 172.21.15.23:0/3840536255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4419f570 con 0x7f9c44103270 2024-03-28T15:52:23.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c48c5c640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c037040 tx=0x7f9c3c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c041020 con 0x7f9c44103270 2024-03-28T15:52:23.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4419f800 con 0x7f9c44103270 2024-03-28T15:52:23.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c3c034cd0 con 0x7f9c44103270 2024-03-28T15:52:23.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c3c039040 con 0x7f9c44103270 2024-03-28T15:52:23.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.088+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4419fcf0 con 0x7f9c44103270 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.089+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c441089e0 con 0x7f9c44103270 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.090+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9c3c044020 con 0x7f9c44103270 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.091+0000 7f9c417fa640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 0x7f9c300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.091+0000 7f9c43fff640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 0x7f9c300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.091+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9c3c0f1020 con 0x7f9c44103270 2024-03-28T15:52:23.095 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.091+0000 7f9c43fff640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 0x7f9c300a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c44102710 tx=0x7f9c34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.099+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9c3c04b020 con 0x7f9c44103270 2024-03-28T15:52:23.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.284+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c4402d240 con 0x7f9c44103270 2024-03-28T15:52:23.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.286+0000 7f9c417fa640 1 -- 172.21.15.23:0/3840536255 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24622 (secure 0 0 0) 0x7f9c3c0bddf0 con 0x7f9c44103270 2024-03-28T15:52:23.287 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:23.287 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":92,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:51:35.345652+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:23.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 msgr2=0x7f9c300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 0x7f9c300a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9c44102710 tx=0x7f9c34005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:23.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 msgr2=0x7f9c4419ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c037040 tx=0x7f9c3c032f40 comp rx=0 tx=0).stop 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 shutdown_connections 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9c300a2800 0x7f9c300a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9c44103270 0x7f9c4419ef00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c441028b0 0x7f9c44197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 --2- 172.21.15.23:0/3840536255 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9c441016c0 0x7f9c44197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 >> 172.21.15.23:0/3840536255 conn(0x7f9c440fcc50 msgr2=0x7f9c44105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 shutdown_connections 2024-03-28T15:52:23.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.288+0000 7f9c4a6bf640 1 -- 172.21.15.23:0/3840536255 wait complete. 2024-03-28T15:52:23.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:22 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/789082816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:22 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3756622442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.300 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:22 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2368275269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.302 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:52:23.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:22 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/789082816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:22 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3756622442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:22 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2368275269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:22 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/789082816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:22 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3756622442' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:22 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2368275269' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- 172.21.15.23:0/531828268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c1021a0 msgr2=0x7fb75c104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 --2- 172.21.15.23:0/531828268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c1021a0 0x7fb75c104590 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb750009900 tx=0x7fb75002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- 172.21.15.23:0/531828268 shutdown_connections 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 --2- 172.21.15.23:0/531828268 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c107400 0x7fb75c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 --2- 172.21.15.23:0/531828268 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c104ad0 0x7fb75c106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 --2- 172.21.15.23:0/531828268 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c1021a0 0x7fb75c104590 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.407 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- 172.21.15.23:0/531828268 >> 172.21.15.23:0/531828268 conn(0x7fb75c0fbb30 msgr2=0x7fb75c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- 172.21.15.23:0/531828268 shutdown_connections 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- 172.21.15.23:0/531828268 wait complete. 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 Processor -- start 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.404+0000 7fb762f97640 1 -- start start 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c104ad0 0x7fb75c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c107400 0x7fb75c19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c10d340 con 0x7fb75c1021a0 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c10d1c0 con 0x7fb75c104ad0 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75c10d4c0 con 0x7fb75c107400 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58226/0 (socket says 172.21.15.23:58226) 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 -- 172.21.15.23:0/35519099 learned_addr learned my addr 172.21.15.23:0/35519099 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.408 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb75bfff640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c104ad0 0x7fb75c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb761534640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c107400 0x7fb75c19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 -- 172.21.15.23:0/35519099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c107400 msgr2=0x7fb75c19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c107400 0x7fb75c19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 -- 172.21.15.23:0/35519099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c104ad0 msgr2=0x7fb75c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c104ad0 0x7fb75c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 -- 172.21.15.23:0/35519099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c19f630 con 0x7fb75c1021a0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb760d33640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb750009230 tx=0x7fb75002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750041070 con 0x7fb75c1021a0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb750004450 con 0x7fb75c1021a0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75c19f8c0 con 0x7fb75c1021a0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.405+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb750003db0 con 0x7fb75c1021a0 2024-03-28T15:52:23.409 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.406+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb75c19fdb0 con 0x7fb75c1021a0 2024-03-28T15:52:23.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.407+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb75c0fd620 con 0x7fb75c1021a0 2024-03-28T15:52:23.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.408+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb750044070 con 0x7fb75c1021a0 2024-03-28T15:52:23.410 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.408+0000 7fb759ffb640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 0x7fb74c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.409+0000 7fb75bfff640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 0x7fb74c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.409+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fb7500f0990 con 0x7fb75c1021a0 2024-03-28T15:52:23.411 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.409+0000 7fb75bfff640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 0x7fb74c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb75c198f20 tx=0x7fb748005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.413+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7500bd6d0 con 0x7fb75c1021a0 2024-03-28T15:52:23.694 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.693+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb75c02d290 con 0x7fb75c1021a0 2024-03-28T15:52:23.695 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.694+0000 7fb759ffb640 1 -- 172.21.15.23:0/35519099 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7fb7500bcfd0 con 0x7fb75c1021a0 2024-03-28T15:52:23.695 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:23.695 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:52:23.695 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 msgr2=0x7fb74c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 0x7fb74c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb75c198f20 tx=0x7fb748005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 msgr2=0x7fb75c197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb750009230 tx=0x7fb75002c650 comp rx=0 tx=0).stop 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 shutdown_connections 2024-03-28T15:52:23.697 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb74c0a2800 0x7fb74c0a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb75c107400 0x7fb75c19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb75c104ad0 0x7fb75c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 --2- 172.21.15.23:0/35519099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb75c1021a0 0x7fb75c197a10 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 >> 172.21.15.23:0/35519099 conn(0x7fb75c0fbb30 msgr2=0x7fb75c102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 shutdown_connections 2024-03-28T15:52:23.698 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.696+0000 7fb762f97640 1 -- 172.21.15.23:0/35519099 wait complete. 2024-03-28T15:52:23.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T15:50:31.785133+0000', 'modified': '2024-03-28T15:51:00.222875+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24494, 'mds_1': 24481, 'mds_2': 24455, 'mds_3': 24454, 'mds_4': 14640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14628': {'gid': 14628, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/1641251810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 1641251810}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 1641251810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14640': {'gid': 14640, 'name': 'a', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.23:6841/910694674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 910694674}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 910694674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24392': {'gid': 24392, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6833/2721990059', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6832', 'nonce': 2721990059}, {'type': 'v1', 'addr': '172.21.15.117:6833', 'nonce': 2721990059}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6833/515874083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6832', 'nonce': 515874083}, {'type': 'v1', 'addr': '172.21.15.120:6833', 'nonce': 515874083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6835/3656482331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 3656482331}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 3656482331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1615812668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1615812668}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1615812668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.120:6837/2763171574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 2763171574}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 2763171574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.117:6837/30575642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 30575642}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 30575642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.120:6839/4092312487', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 4092312487}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 4092312487}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3971000510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3971000510}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3971000510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24481, 24494, 24455, 24454, 14640]} 2024-03-28T15:52:23.709 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T15:52:23.709 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- 172.21.15.23:0/1708323001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c1073a0 msgr2=0x7fe25c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 --2- 172.21.15.23:0/1708323001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c1073a0 0x7fe25c1097f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe258009860 tx=0x7fe25802bb70 comp rx=0 tx=0).stop 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- 172.21.15.23:0/1708323001 shutdown_connections 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 --2- 172.21.15.23:0/1708323001 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c1073a0 0x7fe25c1097f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 --2- 172.21.15.23:0/1708323001 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 0x7fe25c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.817 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 --2- 172.21.15.23:0/1708323001 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c102140 0x7fe25c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- 172.21.15.23:0/1708323001 >> 172.21.15.23:0/1708323001 conn(0x7fe25c0fbb50 msgr2=0x7fe25c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- 172.21.15.23:0/1708323001 shutdown_connections 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- 172.21.15.23:0/1708323001 wait complete. 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 Processor -- start 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.813+0000 7fe263ba9640 1 -- start start 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c102140 0x7fe25c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 0x7fe25c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c10d340 con 0x7fe25c1073a0 2024-03-28T15:52:23.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c10d1c0 con 0x7fe25c102140 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c10d4c0 con 0x7fe25c104a70 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58252/0 (socket says 172.21.15.23:58252) 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 -- 172.21.15.23:0/107715940 learned_addr learned my addr 172.21.15.23:0/107715940 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe261144640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 0x7fe25c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe261945640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c102140 0x7fe25c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 -- 172.21.15.23:0/107715940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 msgr2=0x7fe25c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 0x7fe25c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 -- 172.21.15.23:0/107715940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c102140 msgr2=0x7fe25c197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c102140 0x7fe25c197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 -- 172.21.15.23:0/107715940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c19f5a0 con 0x7fe25c1073a0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe262146640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe258034b10 tx=0x7fe258032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe252ffd640 1 -- 172.21.15.23:0/107715940 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258041030 con 0x7fe25c1073a0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c19f830 con 0x7fe25c1073a0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.814+0000 7fe252ffd640 1 -- 172.21.15.23:0/107715940 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe25802c510 con 0x7fe25c1073a0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.815+0000 7fe252ffd640 1 -- 172.21.15.23:0/107715940 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258039040 con 0x7fe25c1073a0 2024-03-28T15:52:23.820 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.815+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c19fd20 con 0x7fe25c1073a0 2024-03-28T15:52:23.828 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.817+0000 7fe252ffd640 1 -- 172.21.15.23:0/107715940 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe258044070 con 0x7fe25c1073a0 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.817+0000 7fe266793740 1 Processor -- start 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 -- start start 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2d4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2dcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a235c20 con 0x559d0a2d5540 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a1dea40 con 0x559d0a2da6b0 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe266793740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a231cc0 con 0x559d0a2d4b50 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.829 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261144640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2d4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58264/0 (socket says 172.21.15.23:58264) 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 -- 172.21.15.23:0/3335601524 learned_addr learned my addr 172.21.15.23:0/3335601524 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261144640 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2dcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe252ffd640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 -- 172.21.15.23:0/3335601524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 msgr2=0x559d0a2d4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2d4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 -- 172.21.15.23:0/3335601524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 msgr2=0x559d0a2dcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2dcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe261945640 1 -- 172.21.15.23:0/3335601524 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d0a200f90 con 0x559d0a2d5540 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.818+0000 7fe262146640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.830 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.819+0000 7fe252ffd640 1 -- 172.21.15.23:0/107715940 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe2580f0280 con 0x7fe25c1073a0 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.819+0000 7fe261945640 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe254009a60 tx=0x7fe2540095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea0bf2e5c4cd27b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.819+0000 7fe247fff640 1 -- 172.21.15.23:0/3335601524 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25400d8b0 con 0x559d0a2d5540 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.819+0000 7fe247fff640 1 -- 172.21.15.23:0/3335601524 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe25401c070 con 0x559d0a2d5540 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.819+0000 7fe247fff640 1 -- 172.21.15.23:0/3335601524 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254016380 con 0x559d0a2d5540 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- 172.21.15.23:0/3335601524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 msgr2=0x559d0a2da040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe254009a60 tx=0x7fe2540095c0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.831 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe262146640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 0x7fe2480a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe258009830 tx=0x7fe258087680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- 172.21.15.23:0/3335601524 shutdown_connections 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2dcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2da040 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- 172.21.15.23:0/3335601524 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2d4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- 172.21.15.23:0/3335601524 >> 172.21.15.23:0/3335601524 conn(0x559d0a298a60 msgr2=0x559d0a2d04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- 172.21.15.23:0/3335601524 shutdown_connections 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- 172.21.15.23:0/3335601524 wait complete. 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- start start 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 Processor -- start 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2d4b50 0x559d0a2defa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.832 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2df4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2dd660 con 0x559d0a2da6b0 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a1f1e90 con 0x559d0a2d4b50 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.820+0000 7fe266793740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2dd7e0 con 0x559d0a2d5540 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe266793740 10 client.0 _ll_register_callbacks cb 0x559d09f49ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58272/0 (socket says 172.21.15.23:58272) 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 -- 172.21.15.23:0/4224157473 learned_addr learned my addr 172.21.15.23:0/4224157473 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261945640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2df4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.833 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2d4b50 0x559d0a2defa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 -- 172.21.15.23:0/4224157473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 msgr2=0x559d0a2df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2df4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 -- 172.21.15.23:0/4224157473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2d4b50 msgr2=0x559d0a2defa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2d4b50 0x559d0a2defa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 -- 172.21.15.23:0/4224157473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d0a1fe9d0 con 0x559d0a2da6b0 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe261144640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe24c011fd0 tx=0x7fe24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea65d97fe000fd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe2457fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c00fe20 con 0x559d0a2da6b0 2024-03-28T15:52:23.834 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe24c004250 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe25401c070 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.821+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe24c02e050 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 10 client.15219 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d09f4a2d0 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe25401c070 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe24c020040 con 0x559d0a2da6b0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 10 client.15219 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 10 client.15219 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 20 client.15219 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a2eb7f0 0x559d0a2ea860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 4 client.15219 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 --> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d0a243160 con 0x559d0a2eb7f0 2024-03-28T15:52:23.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe261945640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a2eb7f0 0x559d0a2ea860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe261945640 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a2eb7f0 0x559d0a2ea860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b3754808e3d8265 server_cookie=6ddfca4dc737dbd9 in_seq=0 out_seq=0 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.822+0000 7fe2457fa640 10 client.15219 ms_handle_connect on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe2457fa640 1 -- 172.21.15.23:0/4224157473 <== mds.4 v2:172.21.15.23:6840/910694674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d0a243160 con 0x559d0a2eb7f0 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe2457fa640 10 client.15219 handle_command_reply: tid=0 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 client.15219 shutdown 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 msgr2=0x559d0a2e6520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe24c011fd0 tx=0x7fe24c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 shutdown_connections 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a2eb7f0 0x559d0a2ea860 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6520 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe2457fa640 0 client.15219 ms_handle_reset on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2df4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 --2- 172.21.15.23:0/4224157473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2d4b50 0x559d0a2defa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 >> 172.21.15.23:0/4224157473 conn(0x559d0a298a60 msgr2=0x559d0a2cb2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 shutdown_connections 2024-03-28T15:52:23.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 1 -- 172.21.15.23:0/4224157473 wait complete. 2024-03-28T15:52:23.837 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.823+0000 7fe266793740 20 client.15219 trim_cache size 0 max 16384 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 Processor -- start 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 -- start start 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2def70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.849 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2eb010 con 0x559d0a2d5540 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2eb670 con 0x559d0a2da6b0 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe266793740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2ebcd0 con 0x559d0a2d4b50 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe261945640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe261945640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58284/0 (socket says 172.21.15.23:58284) 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe261945640 1 -- 172.21.15.23:0/165493320 learned_addr learned my addr 172.21.15.23:0/165493320 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.840+0000 7fe261144640 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2def70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261144640 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 -- 172.21.15.23:0/165493320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 msgr2=0x559d0a2def70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2def70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 -- 172.21.15.23:0/165493320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 msgr2=0x559d0a2e6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.850 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 -- 172.21.15.23:0/165493320 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d0a0146b0 con 0x559d0a2d5540 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe261945640 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe254009fd0 tx=0x7fe254005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5aab399dd42800d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe2477fe640 1 -- 172.21.15.23:0/165493320 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe254030070 con 0x559d0a2d5540 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe2477fe640 1 -- 172.21.15.23:0/165493320 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe25401c070 con 0x559d0a2d5540 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.841+0000 7fe2477fe640 1 -- 172.21.15.23:0/165493320 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25400da30 con 0x559d0a2d5540 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- 172.21.15.23:0/165493320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 msgr2=0x559d0a2df4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe254009fd0 tx=0x7fe254005bd0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- 172.21.15.23:0/165493320 shutdown_connections 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.851 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d5540 0x559d0a2df4b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 --2- 172.21.15.23:0/165493320 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d4b50 0x559d0a2def70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- 172.21.15.23:0/165493320 >> 172.21.15.23:0/165493320 conn(0x559d0a298a60 msgr2=0x559d0a2d6940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- 172.21.15.23:0/165493320 shutdown_connections 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- 172.21.15.23:0/165493320 wait complete. 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 -- start start 2024-03-28T15:52:23.852 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 Processor -- start 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.842+0000 7fe266793740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2e6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2ec990 con 0x559d0a2d4b50 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2ec330 con 0x559d0a2da6b0 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d0a2ecff0 con 0x559d0a2d5540 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 10 client.0 _ll_register_callbacks cb 0x7fe24c011e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58296/0 (socket says 172.21.15.23:58296) 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 -- 172.21.15.23:0/1720673911 learned_addr learned my addr 172.21.15.23:0/1720673911 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261945640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 -- 172.21.15.23:0/1720673911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 msgr2=0x559d0a2e6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2e6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 -- 172.21.15.23:0/1720673911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 msgr2=0x559d0a2e29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 -- 172.21.15.23:0/1720673911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d0a2e31a0 con 0x559d0a2d4b50 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261945640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe261144640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe24c011e40 tx=0x7fe24c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95d1eb97ba8029 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe2457fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c004460 con 0x559d0a2d4b50 2024-03-28T15:52:23.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe24c011030 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe25401c070 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.843+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe24c011230 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 10 client.15225 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559d0a3475f0 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe25401c070 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe24c003b20 con 0x559d0a2d4b50 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 10 client.15225 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 10 client.15225 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 20 client.15225 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a34bc30 0x559d0a347820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 4 client.15225 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 --> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d0a23d4b0 con 0x559d0a34bc30 2024-03-28T15:52:23.855 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe261945640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a34bc30 0x559d0a347820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe2457fa640 10 client.15225 ms_handle_connect on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.844+0000 7fe261945640 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a34bc30 0x559d0a347820 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2fcbf13cad666550 server_cookie=70d5bfdb7ef883ef in_seq=0 out_seq=0 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe2457fa640 1 -- 172.21.15.23:0/1720673911 <== mds.4 v2:172.21.15.23:6840/910694674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559d0a23d4b0 con 0x559d0a34bc30 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe2457fa640 10 client.15225 handle_command_reply: tid=0 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 client.15225 shutdown 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 msgr2=0x559d0a2e5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe24c011e40 tx=0x7fe24c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 shutdown_connections 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x559d0a34bc30 0x559d0a347820 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559d0a2da6b0 0x559d0a2e29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559d0a2d5540 0x559d0a2e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe2457fa640 0 client.15225 ms_handle_reset on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 --2- 172.21.15.23:0/1720673911 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559d0a2d4b50 0x559d0a2e5f20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 >> 172.21.15.23:0/1720673911 conn(0x559d0a298a60 msgr2=0x559d0a2def70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 shutdown_connections 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 1 -- 172.21.15.23:0/1720673911 wait complete. 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe266793740 20 client.15225 trim_cache size 0 max 16384 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14640, 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:23.857 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 84.627287555999999, 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 118.73264135300001 2024-03-28T15:52:23.858 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 msgr2=0x7fe2480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 0x7fe2480a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe258009830 tx=0x7fe258087680 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 msgr2=0x7fe25c19ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.845+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe258034b10 tx=0x7fe258032470 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 shutdown_connections 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2480a2800 0x7fe2480a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe25c1073a0 0x7fe25c19ef30 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe25c104a70 0x7fe25c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 --2- 172.21.15.23:0/107715940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe25c102140 0x7fe25c197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 >> 172.21.15.23:0/107715940 conn(0x7fe25c0fbb50 msgr2=0x7fe25c102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 shutdown_connections 2024-03-28T15:52:23.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.846+0000 7fe263ba9640 1 -- 172.21.15.23:0/107715940 wait complete. 2024-03-28T15:52:23.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T15:52:23.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 -- 172.21.15.23:0/4100963901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4102120 msgr2=0x7f32e4104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 --2- 172.21.15.23:0/4100963901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4102120 0x7f32e4104510 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009900 tx=0x7f32d402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:23.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 -- 172.21.15.23:0/4100963901 shutdown_connections 2024-03-28T15:52:23.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 --2- 172.21.15.23:0/4100963901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4107380 0x7f32e41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.967 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 --2- 172.21.15.23:0/4100963901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4104a50 0x7f32e4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 --2- 172.21.15.23:0/4100963901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4102120 0x7f32e4104510 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 -- 172.21.15.23:0/4100963901 >> 172.21.15.23:0/4100963901 conn(0x7f32e40fbb30 msgr2=0x7f32e40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 -- 172.21.15.23:0/4100963901 shutdown_connections 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.963+0000 7f32e88f8640 1 -- 172.21.15.23:0/4100963901 wait complete. 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 Processor -- start 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 -- start start 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 0x7f32e4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 0x7f32e419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e410d320 con 0x7f32e4104a50 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e410d1a0 con 0x7f32e4107380 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e88f8640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e410d4a0 con 0x7f32e4102120 2024-03-28T15:52:23.968 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58322/0 (socket says 172.21.15.23:58322) 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3132082100 learned_addr learned my addr 172.21.15.23:0/3132082100 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e2d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 0x7f32e419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.964+0000 7f32e259c640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 0x7f32e4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3132082100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 msgr2=0x7f32e4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 0x7f32e4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3132082100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 msgr2=0x7f32e419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 0x7f32e419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3132082100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e419f670 con 0x7f32e4104a50 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 0x7f32e419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e259c640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 0x7f32e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9a0 tx=0x7f32d800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.969 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32d37fe640 1 -- 172.21.15.23:0/3132082100 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d800cd80 con 0x7f32e4104a50 2024-03-28T15:52:23.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e419f960 con 0x7f32e4104a50 2024-03-28T15:52:23.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32d37fe640 1 -- 172.21.15.23:0/3132082100 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d80035f0 con 0x7f32e4104a50 2024-03-28T15:52:23.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32d37fe640 1 -- 172.21.15.23:0/3132082100 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8010670 con 0x7f32e4104a50 2024-03-28T15:52:23.970 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.965+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e419ff00 con 0x7f32e4104a50 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.967+0000 7f32d37fe640 1 -- 172.21.15.23:0/3132082100 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f32d80037a0 con 0x7f32e4104a50 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 Processor -- start 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 -- start start 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95630b50 0x559b95630f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32d37fe640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 0x7f32cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 0x559b95636040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b95638b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.977 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95591c20 con 0x559b956366b0 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b9553aa40 con 0x559b95631540 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32eb5be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b9558dcc0 con 0x559b95630b50 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e259c640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 0x7f32cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b95638b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e1d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 0x559b95636040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b95638b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58326/0 (socket says 172.21.15.23:58326) 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e2d9d640 1 -- 172.21.15.23:0/2192542746 learned_addr learned my addr 172.21.15.23:0/2192542746 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32d37fe640 1 -- 172.21.15.23:0/3132082100 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f32d80cf020 con 0x7f32e4104a50 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.968+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95630b50 0x559b95630f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 -- 172.21.15.23:0/2192542746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95630b50 msgr2=0x559b95630f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95630b50 0x559b95630f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 -- 172.21.15.23:0/2192542746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 msgr2=0x559b95638b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.978 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b95638b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 -- 172.21.15.23:0/2192542746 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b9555cf90 con 0x559b95631540 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e259c640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 0x7f32cc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f32d402c7b0 tx=0x7f32d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 0x559b95636040 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e970 tx=0x7f32d809b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce0bcc6d995e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32d0ff9640 1 -- 172.21.15.23:0/2192542746 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d80a0cd0 con 0x559b95631540 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32d0ff9640 1 -- 172.21.15.23:0/2192542746 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d809bd60 con 0x559b95631540 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.969+0000 7f32d0ff9640 1 -- 172.21.15.23:0/2192542746 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d80a06d0 con 0x559b95631540 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- 172.21.15.23:0/2192542746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 msgr2=0x559b95636040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 0x559b95636040 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e970 tx=0x7f32d809b4f0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- 172.21.15.23:0/2192542746 shutdown_connections 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b95638b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95631540 0x559b95636040 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2192542746 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95630b50 0x559b95630f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- 172.21.15.23:0/2192542746 >> 172.21.15.23:0/2192542746 conn(0x559b955f4a60 msgr2=0x559b956280b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- 172.21.15.23:0/2192542746 shutdown_connections 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- 172.21.15.23:0/2192542746 wait complete. 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- start start 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 Processor -- start 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 0x559b9563b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b956366b0 0x559b956424d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95639600 con 0x559b95631540 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b9554de90 con 0x559b95630b50 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95639780 con 0x559b956366b0 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.970+0000 7f32eb5be740 10 client.0 _ll_register_callbacks cb 0x559b952a5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 0x559b9563b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.980 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e259c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55176/0 (socket says 172.21.15.23:55176) 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e259c640 1 -- 172.21.15.23:0/3615574489 learned_addr learned my addr 172.21.15.23:0/3615574489 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b956366b0 0x559b956424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b956366b0 msgr2=0x559b956424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b956366b0 0x559b956424d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3615574489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 msgr2=0x559b9563af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 -- 172.21.15.23:0/3615574489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b9555a9d0 con 0x559b95631540 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e259c640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 0x559b9563b490 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f32d809bf10 tx=0x7f32d80a0490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d999fca0b371ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d80b9060 con 0x559b95631540 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32d809bd60 con 0x559b95631540 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f32d809f550 con 0x559b95631540 2024-03-28T15:52:23.981 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d8081050 con 0x559b95631540 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32eb5be740 10 client.15237 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.971+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559b952a62d0 con 0x559b95631540 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32d809bd60 con 0x559b95631540 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f32d80a0cd0 con 0x559b95631540 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 10 client.15237 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 10 client.15237 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 20 client.15237 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b95647890 0x559b95646630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 4 client.15237 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.972+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 --> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b9559f160 con 0x559b95647890 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.973+0000 7f32e259c640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b95647890 0x559b95646630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.973+0000 7f32c9ffb640 10 client.15237 ms_handle_connect on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.973+0000 7f32e259c640 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b95647890 0x559b95646630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5d045c3c1e770500 server_cookie=7ee21a6960e16964 in_seq=0 out_seq=0 2024-03-28T15:52:23.982 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32c9ffb640 1 -- 172.21.15.23:0/3615574489 <== mds.3 v2:172.21.15.120:6836/2763171574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559b9559f160 con 0x559b95647890 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32c9ffb640 10 client.15237 handle_command_reply: tid=0 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 client.15237 shutdown 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 msgr2=0x559b9563b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 0x559b9563b490 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f32d809bf10 tx=0x7f32d80a0490 comp rx=0 tx=0).stop 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 shutdown_connections 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b95647890 0x559b95646630 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b956366b0 0x559b956424d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95631540 0x559b9563b490 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 --2- 172.21.15.23:0/3615574489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b9563af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 >> 172.21.15.23:0/3615574489 conn(0x559b955f4a60 msgr2=0x559b956272f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32c9ffb640 0 client.15237 ms_handle_reset on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 shutdown_connections 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 1 -- 172.21.15.23:0/3615574489 wait complete. 2024-03-28T15:52:23.983 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.974+0000 7f32eb5be740 20 client.15237 trim_cache size 0 max 16384 2024-03-28T15:52:23.993 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 Processor -- start 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 -- start start 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b95638aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b95642490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b956470b0 con 0x559b956366b0 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95647710 con 0x559b95630b50 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.991+0000 7f32eb5be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95647d70 con 0x559b95631540 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e1d9b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b95642490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58348/0 (socket says 172.21.15.23:58348) 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 -- 172.21.15.23:0/648552525 learned_addr learned my addr 172.21.15.23:0/648552525 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e259c640 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b95638aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:23.994 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 -- 172.21.15.23:0/648552525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 msgr2=0x559b95642490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b95642490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 -- 172.21.15.23:0/648552525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 msgr2=0x559b95638aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b95638aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 -- 172.21.15.23:0/648552525 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b953706b0 con 0x559b956366b0 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e259c640 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b95638aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f32dc003ad0 tx=0x7f32dc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21472ad0097b758f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32cbfff640 1 -- 172.21.15.23:0/648552525 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc004780 con 0x559b956366b0 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32cbfff640 1 -- 172.21.15.23:0/648552525 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32dc023070 con 0x559b956366b0 2024-03-28T15:52:23.995 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.992+0000 7f32cbfff640 1 -- 172.21.15.23:0/648552525 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc00f740 con 0x559b956366b0 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- 172.21.15.23:0/648552525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 msgr2=0x559b9562b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f32dc003ad0 tx=0x7f32dc004ab0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- 172.21.15.23:0/648552525 shutdown_connections 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b956366b0 0x559b9562b2f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b95642490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 --2- 172.21.15.23:0/648552525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b95630b50 0x559b95638aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- 172.21.15.23:0/648552525 >> 172.21.15.23:0/648552525 conn(0x559b955f4a60 msgr2=0x559b9563af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- 172.21.15.23:0/648552525 shutdown_connections 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- 172.21.15.23:0/648552525 wait complete. 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 -- start start 2024-03-28T15:52:24.000 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.993+0000 7f32eb5be740 1 Processor -- start 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b9563e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b956366b0 0x559b95641510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95648a30 con 0x559b95630b50 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b956483d0 con 0x559b956366b0 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b95649090 con 0x559b95631540 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 10 client.0 _ll_register_callbacks cb 0x7f32d8097900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58356/0 (socket says 172.21.15.23:58356) 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 -- 172.21.15.23:0/2842817565 learned_addr learned my addr 172.21.15.23:0/2842817565 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b9563e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e2d9d640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b956366b0 0x559b95641510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.001 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 -- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 msgr2=0x559b9563e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b9563e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 -- 172.21.15.23:0/2842817565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b956366b0 msgr2=0x559b95641510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b956366b0 0x559b95641510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 -- 172.21.15.23:0/2842817565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b95641b00 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32e259c640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009230 tx=0x7f32d402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e6f80ee3b3bc88 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d4056030 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f32d4005310 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.994+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32dc023070 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d4063050 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 10 client.15243 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559b956a2d90 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32dc023070 con 0x559b95630b50 2024-03-28T15:52:24.002 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f32d4052070 con 0x559b95630b50 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 10 client.15243 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 10 client.15243 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 20 client.15243 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b956a7530 0x559b956a3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 4 client.15243 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 --> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559b955994b0 con 0x559b956a7530 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.995+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b956a7530 0x559b956a3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32c9ffb640 10 client.15243 ms_handle_connect on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32e1d9b640 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b956a7530 0x559b956a3080 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=80bff3e8f14eca50 server_cookie=7a7c9bf2abd43321 in_seq=0 out_seq=0 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32c9ffb640 1 -- 172.21.15.23:0/2842817565 <== mds.3 v2:172.21.15.120:6836/2763171574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559b955994b0 con 0x559b956a7530 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32c9ffb640 10 client.15243 handle_command_reply: tid=0 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 client.15243 shutdown 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 msgr2=0x559b9563e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009230 tx=0x7f32d402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 shutdown_connections 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x559b956a7530 0x559b956a3080 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b956366b0 0x559b95641510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x559b95631540 0x559b9563e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32c9ffb640 0 client.15243 ms_handle_reset on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 --2- 172.21.15.23:0/2842817565 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x559b95630b50 0x559b9563e000 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 >> 172.21.15.23:0/2842817565 conn(0x559b955f4a60 msgr2=0x559b9562b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 shutdown_connections 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 1 -- 172.21.15.23:0/2842817565 wait complete. 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32eb5be740 20 client.15243 trim_cache size 0 max 16384 2024-03-28T15:52:24.004 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24454, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 85.429919853000001, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 130.22780893999999 2024-03-28T15:52:24.005 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 msgr2=0x7f32cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 0x7f32cc0a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f32d402c7b0 tx=0x7f32d4005fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 msgr2=0x7f32e4197f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.996+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f32d800e9a0 tx=0x7f32d800ee60 comp rx=0 tx=0).stop 2024-03-28T15:52:24.006 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 shutdown_connections 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f32cc0a2800 0x7f32cc0a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e4107380 0x7f32e419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f32e4104a50 0x7f32e4197f90 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 --2- 172.21.15.23:0/3132082100 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f32e4102120 0x7f32e4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 >> 172.21.15.23:0/3132082100 conn(0x7f32e40fbb30 msgr2=0x7f32e4102a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 shutdown_connections 2024-03-28T15:52:24.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:23.997+0000 7f32e88f8640 1 -- 172.21.15.23:0/3132082100 wait complete. 2024-03-28T15:52:24.008 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 -- 172.21.15.23:0/2117720956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c102150 msgr2=0x7fe24c1089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2117720956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c102150 0x7fe24c1089e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe244009ca0 tx=0x7fe24402c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 -- 172.21.15.23:0/2117720956 shutdown_connections 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2117720956 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c102150 0x7fe24c1089e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2117720956 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c101790 0x7fe24c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2117720956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c1005a0 0x7fe24c1009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 -- 172.21.15.23:0/2117720956 >> 172.21.15.23:0/2117720956 conn(0x7fe24c0fbb30 msgr2=0x7fe24c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 -- 172.21.15.23:0/2117720956 shutdown_connections 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.119+0000 7fe2540c6640 1 -- 172.21.15.23:0/2117720956 wait complete. 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 Processor -- start 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 -- start start 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c101790 0x7fe24c197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 0x7fe24c19ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c10c230 con 0x7fe24c1005a0 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c10c0b0 con 0x7fe24c101790 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe2540c6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c10c3b0 con 0x7fe24c102150 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58372/0 (socket says 172.21.15.23:58372) 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 -- 172.21.15.23:0/2882195719 learned_addr learned my addr 172.21.15.23:0/2882195719 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe252663640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 0x7fe24c19ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251661640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c101790 0x7fe24c197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 -- 172.21.15.23:0/2882195719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 msgr2=0x7fe24c19ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 0x7fe24c19ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.120+0000 7fe251e62640 1 -- 172.21.15.23:0/2882195719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c101790 msgr2=0x7fe24c197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe251e62640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c101790 0x7fe24c197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe251e62640 1 -- 172.21.15.23:0/2882195719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c19f5a0 con 0x7fe24c1005a0 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe252663640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 0x7fe24c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe251e62640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe24800e9c0 tx=0x7fe24800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe242ffd640 1 -- 172.21.15.23:0/2882195719 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24800cde0 con 0x7fe24c1005a0 2024-03-28T15:52:24.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe242ffd640 1 -- 172.21.15.23:0/2882195719 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe248010910 con 0x7fe24c1005a0 2024-03-28T15:52:24.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c19f890 con 0x7fe24c1005a0 2024-03-28T15:52:24.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe242ffd640 1 -- 172.21.15.23:0/2882195719 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248018a00 con 0x7fe24c1005a0 2024-03-28T15:52:24.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.121+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c19fe30 con 0x7fe24c1005a0 2024-03-28T15:52:24.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.123+0000 7fe242ffd640 1 -- 172.21.15.23:0/2882195719 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe24801f020 con 0x7fe24c1005a0 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.123+0000 7fe256d93740 1 Processor -- start 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.123+0000 7fe256d93740 1 -- start start 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209deb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe242ffd640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209e6730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec20943c20 con 0x55ec209e42a0 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec208eca40 con 0x55ec209df130 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe256d93740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2093fcc0 con 0x55ec209de740 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251e62640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe252663640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55200/0 (socket says 172.21.15.23:55200) 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 -- 172.21.15.23:0/3453535382 learned_addr learned my addr 172.21.15.23:0/3453535382 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe252663640 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209deb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe242ffd640 1 -- 172.21.15.23:0/2882195719 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe2480cef20 con 0x7fe24c1005a0 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 -- 172.21.15.23:0/3453535382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 msgr2=0x55ec209deb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209deb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 -- 172.21.15.23:0/3453535382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 msgr2=0x55ec209e6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209e6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 -- 172.21.15.23:0/3453535382 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec2090ef90 con 0x55ec209df130 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251e62640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 0x7fe2380a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe24809b4f0 tx=0x7fe24801ff70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.124+0000 7fe251661640 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009c90 tx=0x7fe23c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2c8df7bf86fd26c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe237fff640 1 -- 172.21.15.23:0/3453535382 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e070 con 0x55ec209df130 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe237fff640 1 -- 172.21.15.23:0/3453535382 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe23c009e40 con 0x55ec209df130 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe237fff640 1 -- 172.21.15.23:0/3453535382 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c018830 con 0x55ec209df130 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 -- 172.21.15.23:0/3453535382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 msgr2=0x55ec209e3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009c90 tx=0x7fe23c011810 comp rx=0 tx=0).stop 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 -- 172.21.15.23:0/3453535382 shutdown_connections 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209e6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209e3c30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 --2- 172.21.15.23:0/3453535382 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209deb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 -- 172.21.15.23:0/3453535382 >> 172.21.15.23:0/3453535382 conn(0x55ec209a6a60 msgr2=0x55ec209da0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 -- 172.21.15.23:0/3453535382 shutdown_connections 2024-03-28T15:52:24.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.125+0000 7fe256d93740 1 -- 172.21.15.23:0/3453535382 wait complete. 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 -- start start 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 Processor -- start 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209f0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209e7250 con 0x55ec209df130 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec208ffe90 con 0x55ec209e42a0 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209e73d0 con 0x55ec209de740 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe251661640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe256d93740 10 client.0 _ll_register_callbacks cb 0x55ec20657ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe251661640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58382/0 (socket says 172.21.15.23:58382) 2024-03-28T15:52:24.136 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe251661640 1 -- 172.21.15.23:0/965471379 learned_addr learned my addr 172.21.15.23:0/965471379 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe252663640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209f0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.126+0000 7fe252663640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 -- 172.21.15.23:0/965471379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 msgr2=0x55ec209e8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 -- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 msgr2=0x55ec209f0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209f0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 -- 172.21.15.23:0/965471379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec2090c9d0 con 0x55ec209df130 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe251661640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0058c0 tx=0x7fe23c005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da082078aefabe1d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e040 con 0x55ec209df130 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe23c004030 con 0x55ec209df130 2024-03-28T15:52:24.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec20951160 con 0x55ec209df130 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe23c032050 con 0x55ec209df130 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe256d93740 10 client.15255 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ec206582d0 con 0x55ec209df130 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ec20951160 con 0x55ec209df130 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe23c021030 con 0x55ec209df130 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.127+0000 7fe256d93740 10 client.15255 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe256d93740 10 client.15255 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe256d93740 20 client.15255 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec209f5180 0x55ec209f4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe256d93740 4 client.15255 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 --> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec2094b4b0 con 0x55ec209f5180 2024-03-28T15:52:24.138 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe252663640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec209f5180 0x55ec209f4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe2357fa640 10 client.15255 ms_handle_connect on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.128+0000 7fe252663640 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec209f5180 0x55ec209f4410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f17e2860dd2518e3 server_cookie=173808cee2b4d24e in_seq=0 out_seq=0 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe2357fa640 1 -- 172.21.15.23:0/965471379 <== mds.2 v2:172.21.15.117:6836/30575642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ec2094b4b0 con 0x55ec209f5180 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe2357fa640 10 client.15255 handle_command_reply: tid=0 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 client.15255 shutdown 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 msgr2=0x55ec209e91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0058c0 tx=0x7fe23c005720 comp rx=0 tx=0).stop 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 shutdown_connections 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec209f5180 0x55ec209f4410 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209f0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209e91c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 --2- 172.21.15.23:0/965471379 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe2357fa640 0 client.15255 ms_handle_reset on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:24.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 >> 172.21.15.23:0/965471379 conn(0x55ec209a6a60 msgr2=0x55ec209d92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 shutdown_connections 2024-03-28T15:52:24.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 1 -- 172.21.15.23:0/965471379 wait complete. 2024-03-28T15:52:24.140 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.129+0000 7fe256d93740 20 client.15255 trim_cache size 0 max 16384 2024-03-28T15:52:24.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 Processor -- start 2024-03-28T15:52:24.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 -- start start 2024-03-28T15:52:24.153 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f49a0 con 0x55ec209df130 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f5000 con 0x55ec209e42a0 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe256d93740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f5660 con 0x55ec209de740 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58384/0 (socket says 172.21.15.23:58384) 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe252663640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 -- 172.21.15.23:0/2432158532 learned_addr learned my addr 172.21.15.23:0/2432158532 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe252663640 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 -- 172.21.15.23:0/2432158532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 msgr2=0x55ec209e8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.154 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 -- 172.21.15.23:0/2432158532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 msgr2=0x55ec209e8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209e8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.146+0000 7fe251661640 1 -- 172.21.15.23:0/2432158532 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec207226b0 con 0x55ec209df130 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe251661640 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe23c012fd0 tx=0x7fe23c009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ee446643e197df server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe2377fe640 1 -- 172.21.15.23:0/2432158532 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e040 con 0x55ec209df130 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe2377fe640 1 -- 172.21.15.23:0/2432158532 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe23c005d30 con 0x55ec209df130 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe2377fe640 1 -- 172.21.15.23:0/2432158532 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c024680 con 0x55ec209df130 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe256d93740 1 -- 172.21.15.23:0/2432158532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 msgr2=0x55ec209f01c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.147+0000 7fe256d93740 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe23c012fd0 tx=0x7fe23c009d80 comp rx=0 tx=0).stop 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 -- 172.21.15.23:0/2432158532 shutdown_connections 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209e42a0 0x55ec209e8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.155 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209df130 0x55ec209f01c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 --2- 172.21.15.23:0/2432158532 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209e8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 -- 172.21.15.23:0/2432158532 >> 172.21.15.23:0/2432158532 conn(0x55ec209a6a60 msgr2=0x55ec209e7770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 -- 172.21.15.23:0/2432158532 shutdown_connections 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 -- 172.21.15.23:0/2432158532 wait complete. 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 -- start start 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 Processor -- start 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209ec070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.148+0000 7fe256d93740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f6320 con 0x55ec209e42a0 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f5cc0 con 0x55ec209df130 2024-03-28T15:52:24.156 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec209f6980 con 0x55ec209de740 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 10 client.0 _ll_register_callbacks cb 0x7fe23c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe252663640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55210/0 (socket says 172.21.15.23:55210) 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 -- 172.21.15.23:0/671531690 learned_addr learned my addr 172.21.15.23:0/671531690 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe252663640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209ec070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 -- 172.21.15.23:0/671531690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 msgr2=0x55ec209ec070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209ec070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 -- 172.21.15.23:0/671531690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 msgr2=0x55ec209ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 -- 172.21.15.23:0/671531690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec209efb00 con 0x55ec209df130 2024-03-28T15:52:24.157 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe252663640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe251661640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe23c01b040 tx=0x7fe23c0096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87fd5325f0cc278 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe2357fa640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c01e070 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe23c005d30 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fe23c0037f0 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe23c044050 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.149+0000 7fe256d93740 10 client.25106 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ec20a508e0 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe23c005d30 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fe23c021070 con 0x55ec209df130 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 10 client.25106 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.158 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 10 client.25106 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 20 client.25106 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec20a54c10 0x55ec20a50b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 4 client.25106 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 --> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec208c99a0 con 0x55ec20a54c10 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe252663640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec20a54c10 0x55ec20a50b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe252663640 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec20a54c10 0x55ec20a50b10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5864a06d8b2fc96b server_cookie=824ab9f266c2e724 in_seq=0 out_seq=0 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.150+0000 7fe2357fa640 10 client.25106 ms_handle_connect on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2357fa640 1 -- 172.21.15.23:0/671531690 <== mds.2 v2:172.21.15.117:6836/30575642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55ec208c99a0 con 0x55ec20a54c10 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2357fa640 10 client.25106 handle_command_reply: tid=0 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 client.25106 shutdown 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 msgr2=0x55ec209ec5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.159 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe23c01b040 tx=0x7fe23c0096c0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 shutdown_connections 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55ec20a54c10 0x55ec20a50b10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ec209e42a0 0x55ec209ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ec209df130 0x55ec209ec5b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 --2- 172.21.15.23:0/671531690 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ec209de740 0x55ec209ec070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2357fa640 0 client.25106 ms_handle_reset on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 >> 172.21.15.23:0/671531690 conn(0x55ec209a6a60 msgr2=0x55ec209e8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 shutdown_connections 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 1 -- 172.21.15.23:0/671531690 wait complete. 2024-03-28T15:52:24.160 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe256d93740 20 client.25106 trim_cache size 0 max 16384 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24455, 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 86.594060747, 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-28T15:52:24.161 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:24.162 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 127.463260596 2024-03-28T15:52:24.162 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:24.162 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 msgr2=0x7fe2380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 0x7fe2380a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe24809b4f0 tx=0x7fe24801ff70 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 msgr2=0x7fe24c197980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe24800e9c0 tx=0x7fe24800ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 shutdown_connections 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fe2380a2880 0x7fe2380a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fe24c102150 0x7fe24c19ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe24c101790 0x7fe24c197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 --2- 172.21.15.23:0/2882195719 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fe24c1005a0 0x7fe24c197980 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.151+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 >> 172.21.15.23:0/2882195719 conn(0x7fe24c0fbb30 msgr2=0x7fe24c0fd5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.152+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 shutdown_connections 2024-03-28T15:52:24.163 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.152+0000 7fe2540c6640 1 -- 172.21.15.23:0/2882195719 wait complete. 2024-03-28T15:52:24.164 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T15:52:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 -- 172.21.15.23:0/236507552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01016c0 msgr2=0x7f57e0101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.276 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 --2- 172.21.15.23:0/236507552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01016c0 0x7f57e0101ae0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009900 tx=0x7f57d002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 -- 172.21.15.23:0/236507552 shutdown_connections 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 --2- 172.21.15.23:0/236507552 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 0x7f57e0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 --2- 172.21.15.23:0/236507552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01028b0 0x7f57e0102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 --2- 172.21.15.23:0/236507552 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01016c0 0x7f57e0101ae0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 -- 172.21.15.23:0/236507552 >> 172.21.15.23:0/236507552 conn(0x7f57e00fcc30 msgr2=0x7f57e00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 -- 172.21.15.23:0/236507552 shutdown_connections 2024-03-28T15:52:24.277 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.275+0000 7f57e69bc640 1 -- 172.21.15.23:0/236507552 wait complete. 2024-03-28T15:52:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 Processor -- start 2024-03-28T15:52:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 -- start start 2024-03-28T15:52:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01016c0 0x7f57e0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 0x7f57e019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e010d360 con 0x7f57e01028b0 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e010d1e0 con 0x7f57e01016c0 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e69bc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e010d4e0 con 0x7f57e0103270 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57df7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57df7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58406/0 (socket says 172.21.15.23:58406) 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57df7fe640 1 -- 172.21.15.23:0/1543740164 learned_addr learned my addr 172.21.15.23:0/1543740164 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57e4f59640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 0x7f57e019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.276+0000 7f57dffff640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01016c0 0x7f57e0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 -- 172.21.15.23:0/1543740164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 msgr2=0x7f57e019efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 0x7f57e019efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 -- 172.21.15.23:0/1543740164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01016c0 msgr2=0x7f57e0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01016c0 0x7f57e0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 -- 172.21.15.23:0/1543740164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e019f650 con 0x7f57e01028b0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57df7fe640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57d400ea60 tx=0x7f57d400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57dd7fa640 1 -- 172.21.15.23:0/1543740164 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d400cd20 con 0x7f57e01028b0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e019f940 con 0x7f57e01028b0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57dd7fa640 1 -- 172.21.15.23:0/1543740164 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d40035f0 con 0x7f57e01028b0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57dd7fa640 1 -- 172.21.15.23:0/1543740164 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d4010690 con 0x7f57e01028b0 2024-03-28T15:52:24.281 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.277+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e019fee0 con 0x7f57e01028b0 2024-03-28T15:52:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.279+0000 7f57dd7fa640 1 -- 172.21.15.23:0/1543740164 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f57d40037a0 con 0x7f57e01028b0 2024-03-28T15:52:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.279+0000 7f57e959c740 1 Processor -- start 2024-03-28T15:52:24.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.279+0000 7f57e959c740 1 -- start start 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.279+0000 7f57e959c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e959c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8522c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e959c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c8525730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e959c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c8482c20 con 0x55f4c851d740 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e959c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c842ba40 con 0x55f4c851e130 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e959c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c847ecc0 con 0x55f4c85232a0 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58418/0 (socket says 172.21.15.23:58418) 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8522c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e4f59640 1 -- 172.21.15.23:0/2638041420 learned_addr learned my addr 172.21.15.23:0/2638041420 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57df7fe640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c8525730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 -- 172.21.15.23:0/2638041420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 msgr2=0x55f4c8525730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c8525730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 -- 172.21.15.23:0/2638041420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 msgr2=0x55f4c851db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 -- 172.21.15.23:0/2638041420 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4c844df90 con 0x55f4c851e130 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e4f59640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dd7fa640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 0x7f57c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57df7fe640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c8525730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57e4f59640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 0x7f57c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dd7fa640 1 -- 172.21.15.23:0/1543740164 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f57d40cf0c0 con 0x7f57e01028b0 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57dffff640 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8522c30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009230 tx=0x7f57d002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d945f64b1b4543af server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57c67fc640 1 -- 172.21.15.23:0/2638041420 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d002c980 con 0x55f4c851e130 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57c67fc640 1 -- 172.21.15.23:0/2638041420 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d002cb20 con 0x55f4c851e130 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.280+0000 7f57c67fc640 1 -- 172.21.15.23:0/2638041420 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0004920 con 0x55f4c851e130 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 -- 172.21.15.23:0/2638041420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 msgr2=0x55f4c8522c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e4f59640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 0x7f57c80a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55f4c851e990 tx=0x7f57d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8522c30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009230 tx=0x7f57d002c670 comp rx=0 tx=0).stop 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 -- 172.21.15.23:0/2638041420 shutdown_connections 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c8525730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8522c30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 --2- 172.21.15.23:0/2638041420 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851d740 0x55f4c851db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 -- 172.21.15.23:0/2638041420 >> 172.21.15.23:0/2638041420 conn(0x55f4c84e5a60 msgr2=0x55f4c85190b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 -- 172.21.15.23:0/2638041420 shutdown_connections 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.281+0000 7f57e959c740 1 -- 172.21.15.23:0/2638041420 wait complete. 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 -- start start 2024-03-28T15:52:24.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 Processor -- start 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8528190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c852f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c85261f0 con 0x55f4c85232a0 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c843ee90 con 0x55f4c851e130 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c8526370 con 0x55f4c851d740 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e959c740 10 client.0 _ll_register_callbacks cb 0x55f4c8196ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c852f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57df7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8528190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:45202/0 (socket says 172.21.15.23:45202) 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 -- 172.21.15.23:0/1217386473 learned_addr learned my addr 172.21.15.23:0/1217386473 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c852f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58422/0 (socket says 172.21.15.23:58422) 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 -- 172.21.15.23:0/1217386473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 msgr2=0x55f4c8528190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8528190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 -- 172.21.15.23:0/1217386473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 msgr2=0x55f4c852f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c852f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.282+0000 7f57dffff640 1 -- 172.21.15.23:0/1217386473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4c844b9d0 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57dffff640 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009900 tx=0x7f57d002bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55deec69656bb1ad server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0050070 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57d002cb20 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f57d0003a00 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57d005c050 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 10 client.25066 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f4c81972d0 con 0x55f4c851d740 2024-03-28T15:52:24.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d002cb20 con 0x55f4c851d740 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f57d003fac0 con 0x55f4c851d740 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 10 client.25066 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 10 client.25066 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 20 client.25066 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c8533330 0x55f4c8538450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.283+0000 7f57e959c740 4 client.25066 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.284+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 --> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4c8490160 con 0x55f4c8533330 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.284+0000 7f57df7fe640 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c8533330 0x55f4c8538450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.284+0000 7f57bf7fe640 10 client.25066 ms_handle_connect on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.284+0000 7f57df7fe640 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c8533330 0x55f4c8538450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=828a9f5a2b9f6f76 server_cookie=dce78c5ed7c6a38b in_seq=0 out_seq=0 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57bf7fe640 1 -- 172.21.15.23:0/1217386473 <== mds.1 v2:172.21.15.120:6838/4092312487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f4c8490160 con 0x55f4c8533330 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57bf7fe640 10 client.25066 handle_command_reply: tid=0 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 client.25066 shutdown 2024-03-28T15:52:24.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 msgr2=0x55f4c8527c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009900 tx=0x7f57d002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 shutdown_connections 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c8533330 0x55f4c8538450 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c852f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c8528190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 --2- 172.21.15.23:0/1217386473 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c8527c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57bf7fe640 0 client.25066 ms_handle_reset on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 >> 172.21.15.23:0/1217386473 conn(0x55f4c84e5a60 msgr2=0x55f4c85182f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 shutdown_connections 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 1 -- 172.21.15.23:0/1217386473 wait complete. 2024-03-28T15:52:24.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.285+0000 7f57e959c740 20 client.25066 trim_cache size 0 max 16384 2024-03-28T15:52:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:23 smithi117 ceph-mon[36002]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:23 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1022178021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:23 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3840536255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:23 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/35519099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 Processor -- start 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 -- start start 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c851ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c852f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c85337f0 con 0x55f4c85232a0 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c8533e50 con 0x55f4c851e130 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.314+0000 7f57e959c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c85344b0 con 0x55f4c851d740 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58432/0 (socket says 172.21.15.23:58432) 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c851ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57e4f59640 1 -- 172.21.15.23:0/3507717375 learned_addr learned my addr 172.21.15.23:0/3507717375 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.322 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57df7fe640 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c852f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 -- 172.21.15.23:0/3507717375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 msgr2=0x55f4c852f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c852f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 -- 172.21.15.23:0/3507717375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 msgr2=0x55f4c853c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 -- 172.21.15.23:0/3507717375 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4c8528d80 con 0x55f4c851d740 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57e4f59640 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57dffff640 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c851ca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f57d0005c40 tx=0x7f57d002c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd60129d074f1b89 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57c5ffb640 1 -- 172.21.15.23:0/3507717375 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0059020 con 0x55f4c851d740 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57c5ffb640 1 -- 172.21.15.23:0/3507717375 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d0034440 con 0x55f4c851d740 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.315+0000 7f57c5ffb640 1 -- 172.21.15.23:0/3507717375 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d0050070 con 0x55f4c851d740 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- 172.21.15.23:0/3507717375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 msgr2=0x55f4c851ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c851ca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f57d0005c40 tx=0x7f57d002c670 comp rx=0 tx=0).stop 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- 172.21.15.23:0/3507717375 shutdown_connections 2024-03-28T15:52:24.323 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c85232a0 0x55f4c853c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851e130 0x55f4c852f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 --2- 172.21.15.23:0/3507717375 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c851d740 0x55f4c851ca00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- 172.21.15.23:0/3507717375 >> 172.21.15.23:0/3507717375 conn(0x55f4c84e5a60 msgr2=0x55f4c851c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- 172.21.15.23:0/3507717375 shutdown_connections 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- 172.21.15.23:0/3507717375 wait complete. 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.316+0000 7f57e959c740 1 -- start start 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 Processor -- start 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c852b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c8535170 con 0x55f4c851e130 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c8534b10 con 0x55f4c851d740 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4c85357d0 con 0x55f4c85232a0 2024-03-28T15:52:24.324 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e959c740 10 client.0 _ll_register_callbacks cb 0x7f57d8009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57dffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58448/0 (socket says 172.21.15.23:58448) 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57e4f59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c852b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 -- 172.21.15.23:0/560812385 learned_addr learned my addr 172.21.15.23:0/560812385 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57dffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:55274/0 (socket says 172.21.15.23:55274) 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 -- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 msgr2=0x55f4c852b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c852b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 -- 172.21.15.23:0/560812385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 msgr2=0x55f4c852ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57df7fe640 1 -- 172.21.15.23:0/560812385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f4c82616b0 con 0x55f4c851e130 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.317+0000 7f57dffff640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.325 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57df7fe640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57d401f000 tx=0x7f57d40cc730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab74f1b8e782478a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57d40b3020 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f57d4099a00 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f57d0034440 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f57d409b060 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 10 client.15279 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f4c85902d0 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f57d0034440 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f57d4097d70 con 0x55f4c851e130 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 10 client.15279 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 10 client.15279 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 20 client.15279 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c85948e0 0x55f4c85905c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.326 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 4 client.15279 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.318+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 --> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55f4c848a4b0 con 0x55f4c85948e0 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57dffff640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c85948e0 0x55f4c85905c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57bf7fe640 10 client.15279 ms_handle_connect on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57dffff640 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c85948e0 0x55f4c85905c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=850392af471d19c server_cookie=94ffd763fd2ed349 in_seq=0 out_seq=0 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57bf7fe640 1 -- 172.21.15.23:0/560812385 <== mds.1 v2:172.21.15.120:6838/4092312487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55f4c848a4b0 con 0x55f4c85948e0 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57bf7fe640 10 client.15279 handle_command_reply: tid=0 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 client.15279 shutdown 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 msgr2=0x55f4c852f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f57d401f000 tx=0x7f57d40cc730 comp rx=0 tx=0).stop 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 shutdown_connections 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55f4c85232a0 0x55f4c852b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x55f4c85948e0 0x55f4c85905c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55f4c851e130 0x55f4c852f180 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 --2- 172.21.15.23:0/560812385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f4c851d740 0x55f4c852ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.327 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57bf7fe640 0 client.15279 ms_handle_reset on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.319+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 >> 172.21.15.23:0/560812385 conn(0x55f4c84e5a60 msgr2=0x55f4c8517a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 shutdown_connections 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e959c740 1 -- 172.21.15.23:0/560812385 wait complete. 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e959c740 20 client.15279 trim_cache size 0 max 16384 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24481, 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:24.328 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 87.768138703000005, 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 124.749891194 2024-03-28T15:52:24.329 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 msgr2=0x7f57c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 0x7f57c80a4cc0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55f4c851e990 tx=0x7f57d8009500 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 msgr2=0x7f57e0197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57d400ea60 tx=0x7f57d400ee60 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 shutdown_connections 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f57c80a2800 0x7f57c80a4cc0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f57e0103270 0x7f57e019efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f57e01028b0 0x7f57e0197f70 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 --2- 172.21.15.23:0/1543740164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e01016c0 0x7f57e0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 >> 172.21.15.23:0/1543740164 conn(0x7f57e00fcc30 msgr2=0x7f57e0105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.330 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 shutdown_connections 2024-03-28T15:52:24.331 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.320+0000 7f57e69bc640 1 -- 172.21.15.23:0/1543740164 wait complete. 2024-03-28T15:52:24.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:23 smithi023 ceph-mon[32448]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:24.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:23 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1022178021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:23 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3840536255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.331 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:23 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/35519099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.332 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T15:52:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:23 smithi120 ceph-mon[36826]: pgmap v346: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:23 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1022178021' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:23 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3840536255' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:23 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/35519099' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:24.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 -- 172.21.15.23:0/2084226004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82801028a0 msgr2=0x7f8280102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 --2- 172.21.15.23:0/2084226004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82801028a0 0x7f8280102d20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8270009900 tx=0x7f827002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 -- 172.21.15.23:0/2084226004 shutdown_connections 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 --2- 172.21.15.23:0/2084226004 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8280103260 0x7f8280109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 --2- 172.21.15.23:0/2084226004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82801028a0 0x7f8280102d20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 --2- 172.21.15.23:0/2084226004 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801016b0 0x7f8280101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 -- 172.21.15.23:0/2084226004 >> 172.21.15.23:0/2084226004 conn(0x7f82800fcc00 msgr2=0x7f82800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 -- 172.21.15.23:0/2084226004 shutdown_connections 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.437+0000 7f8287e59640 1 -- 172.21.15.23:0/2084226004 wait complete. 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 Processor -- start 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 -- start start 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 0x7f8280073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8280103260 0x7f8280073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828010d310 con 0x7f82801016b0 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828010d190 con 0x7f8280103260 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8287e59640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828010d490 con 0x7f82801028a0 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8285bf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8285bf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58462/0 (socket says 172.21.15.23:58462) 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f82853f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 0x7f8280073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f8285bf5640 1 -- 172.21.15.23:0/2126973811 learned_addr learned my addr 172.21.15.23:0/2126973811 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.438+0000 7f82863f6640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8280103260 0x7f8280073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 -- 172.21.15.23:0/2126973811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 msgr2=0x7f8280073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 0x7f8280073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 -- 172.21.15.23:0/2126973811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8280103260 msgr2=0x7f8280073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8280103260 0x7f8280073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 -- 172.21.15.23:0/2126973811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280074860 con 0x7f82801016b0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f82853f4640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 0x7f8280073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9c0 tx=0x7f827c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8276ffd640 1 -- 172.21.15.23:0/2126973811 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c00cd80 con 0x7f82801016b0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82801a1b30 con 0x7f82801016b0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8276ffd640 1 -- 172.21.15.23:0/2126973811 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f827c0035f0 con 0x7f82801016b0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8276ffd640 1 -- 172.21.15.23:0/2126973811 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c010690 con 0x7f82801016b0 2024-03-28T15:52:24.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.439+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82801a2050 con 0x7f82801016b0 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.441+0000 7f8276ffd640 1 -- 172.21.15.23:0/2126973811 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f827c0037a0 con 0x7f82801016b0 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 Processor -- start 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 -- start start 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c695b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c69ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c69d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c5fac20 con 0x55c10c695740 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c5a3a40 con 0x55c10c69b2a0 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f828aa42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c5f6cc0 con 0x55c10c696130 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c695b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f82853f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c69d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c695b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58476/0 (socket says 172.21.15.23:58476) 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f8285bf5640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c69ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f82863f6640 1 -- 172.21.15.23:0/1318139021 learned_addr learned my addr 172.21.15.23:0/1318139021 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.442+0000 7f8276ffd640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 0x7f826c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82863f6640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 0x7f826c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 -- 172.21.15.23:0/1318139021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 msgr2=0x55c10c69ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c69ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 -- 172.21.15.23:0/1318139021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 msgr2=0x55c10c695b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c695b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 -- 172.21.15.23:0/1318139021 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c10c5c5f90 con 0x55c10c69b2a0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f8276ffd640 1 -- 172.21.15.23:0/2126973811 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f827c0cf1b0 con 0x7f82801016b0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82853f4640 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c69d730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8270009230 tx=0x7f827002c6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d70e664288a2188 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f82863f6640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 0x7f826c0a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55c10c696990 tx=0x7f8278004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f826bfff640 1 -- 172.21.15.23:0/1318139021 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270005d50 con 0x55c10c69b2a0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f826bfff640 1 -- 172.21.15.23:0/1318139021 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8270005520 con 0x55c10c69b2a0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.443+0000 7f826bfff640 1 -- 172.21.15.23:0/1318139021 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8270004c20 con 0x55c10c69b2a0 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- 172.21.15.23:0/1318139021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 msgr2=0x55c10c69d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c69d730 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8270009230 tx=0x7f827002c6d0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- 172.21.15.23:0/1318139021 shutdown_connections 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c69d730 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c69ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 --2- 172.21.15.23:0/1318139021 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c695b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- 172.21.15.23:0/1318139021 >> 172.21.15.23:0/1318139021 conn(0x55c10c65da60 msgr2=0x55c10c6910b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.455 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- 172.21.15.23:0/1318139021 shutdown_connections 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- 172.21.15.23:0/1318139021 wait complete. 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 -- start start 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.444+0000 7f828aa42740 1 Processor -- start 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c695740 0x55c10c69fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 0x55c10c6a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c69e1f0 con 0x55c10c69b2a0 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c5b6e90 con 0x55c10c696130 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c69e370 con 0x55c10c695740 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f828aa42740 10 client.0 _ll_register_callbacks cb 0x55c10c30eac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58482/0 (socket says 172.21.15.23:58482) 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 -- 172.21.15.23:0/701090796 learned_addr learned my addr 172.21.15.23:0/701090796 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82853f4640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 0x55c10c6a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f8285bf5640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c695740 0x55c10c69fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 -- 172.21.15.23:0/701090796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c695740 msgr2=0x55c10c69fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c695740 0x55c10c69fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 -- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 msgr2=0x55c10c6a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 0x55c10c6a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 -- 172.21.15.23:0/701090796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c10c5c39d0 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82853f4640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 0x55c10c6a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82863f6640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8278004250 tx=0x7f8278004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabeabb2ecfcf713 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.445+0000 7f82697fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278026070 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f8278004d20 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8270005520 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8278031050 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 10 client.15291 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c10c30f2d0 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8270005520 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f82780056c0 con 0x55c10c69b2a0 2024-03-28T15:52:24.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 10 client.15291 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 10 client.15291 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 20 client.15291 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c6ac440 0x55c10c6ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 4 client.15291 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c10c608160 con 0x55c10c6ac440 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.446+0000 7f8285bf5640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c6ac440 0x55c10c6ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.447+0000 7f82697fa640 10 client.15291 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.447+0000 7f8285bf5640 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c6ac440 0x55c10c6ab4b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4d4c79547a64a75 server_cookie=7bb0e3e4c5c1d651 in_seq=0 out_seq=0 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.447+0000 7f82697fa640 1 -- 172.21.15.23:0/701090796 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c10c608160 con 0x55c10c6ac440 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f82697fa640 10 client.15291 handle_command_reply: tid=0 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 client.15291 shutdown 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 msgr2=0x55c10c6a71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f8278004250 tx=0x7f8278004a40 comp rx=0 tx=0).stop 2024-03-28T15:52:24.459 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 shutdown_connections 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c6ac440 0x55c10c6ab4b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a71c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c696130 0x55c10c6a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f82697fa640 0 client.15291 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 --2- 172.21.15.23:0/701090796 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c695740 0x55c10c69fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 >> 172.21.15.23:0/701090796 conn(0x55c10c65da60 msgr2=0x55c10c6902f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 shutdown_connections 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 1 -- 172.21.15.23:0/701090796 wait complete. 2024-03-28T15:52:24.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.448+0000 7f828aa42740 20 client.15291 trim_cache size 0 max 16384 2024-03-28T15:52:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 Processor -- start 2024-03-28T15:52:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 -- start start 2024-03-28T15:52:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c695740 0x55c10c6976d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c697c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6abc60 con 0x55c10c69b2a0 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6ac2c0 con 0x55c10c695740 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f828aa42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6ac920 con 0x55c10c696130 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58496/0 (socket says 172.21.15.23:58496) 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 -- 172.21.15.23:0/474760966 learned_addr learned my addr 172.21.15.23:0/474760966 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f8285bf5640 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c695740 0x55c10c6976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82853f4640 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c697c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 -- 172.21.15.23:0/474760966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 msgr2=0x55c10c697c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c697c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 -- 172.21.15.23:0/474760966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c695740 msgr2=0x55c10c6976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.472 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c695740 0x55c10c6976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 -- 172.21.15.23:0/474760966 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c10c3d96b0 con 0x55c10c69b2a0 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.465+0000 7f82863f6640 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8278006010 tx=0x7f8278000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72dcd74c36041fae server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f826b7fe640 1 -- 172.21.15.23:0/474760966 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278026050 con 0x55c10c69b2a0 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f826b7fe640 1 -- 172.21.15.23:0/474760966 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f827803a070 con 0x55c10c69b2a0 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f826b7fe640 1 -- 172.21.15.23:0/474760966 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8278035690 con 0x55c10c69b2a0 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- 172.21.15.23:0/474760966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 msgr2=0x55c10c6a7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8278006010 tx=0x7f8278000f40 comp rx=0 tx=0).stop 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- 172.21.15.23:0/474760966 shutdown_connections 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c69b2a0 0x55c10c6a7180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c697c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 --2- 172.21.15.23:0/474760966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c695740 0x55c10c6976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- 172.21.15.23:0/474760966 >> 172.21.15.23:0/474760966 conn(0x55c10c65da60 msgr2=0x55c10c696b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- 172.21.15.23:0/474760966 shutdown_connections 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- 172.21.15.23:0/474760966 wait complete. 2024-03-28T15:52:24.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.466+0000 7f828aa42740 1 -- start start 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 Processor -- start 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c6a3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c6a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6ad5e0 con 0x55c10c695740 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6acf80 con 0x55c10c69b2a0 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c10c6adc40 con 0x55c10c696130 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f828aa42740 10 client.0 _ll_register_callbacks cb 0x7f8278013ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58502/0 (socket says 172.21.15.23:58502) 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 -- 172.21.15.23:0/2510074612 learned_addr learned my addr 172.21.15.23:0/2510074612 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f82853f4640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c6a3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f82863f6640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c6a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.474 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 -- 172.21.15.23:0/2510074612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 msgr2=0x55c10c6a3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c6a3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 -- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 msgr2=0x55c10c6a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c6a63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 -- 172.21.15.23:0/2510074612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c10c6a69a0 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f8285bf5640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f827c0cccc0 tx=0x7f827c06d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d386e08329f3c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.467+0000 7f82697fa640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827c09bef0 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f827c003b30 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f827803a070 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f827c09b7f0 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 10 client.15297 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c10c707ab0 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f827803a070 con 0x55c10c695740 2024-03-28T15:52:24.475 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f827c0bb040 con 0x55c10c695740 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 10 client.15297 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 10 client.15297 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 20 client.15297 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c70c110 0x55c10c707da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 4 client.15297 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c10c6024b0 con 0x55c10c70c110 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.468+0000 7f82853f4640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c70c110 0x55c10c707da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f82697fa640 10 client.15297 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f82853f4640 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c70c110 0x55c10c707da0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f1641319fcd390f server_cookie=4a036214a02c29f1 in_seq=0 out_seq=0 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f82697fa640 1 -- 172.21.15.23:0/2510074612 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55c10c6024b0 con 0x55c10c70c110 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f82697fa640 10 client.15297 handle_command_reply: tid=0 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 client.15297 shutdown 2024-03-28T15:52:24.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 msgr2=0x55c10c6a2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f827c0cccc0 tx=0x7f827c06d040 comp rx=0 tx=0).stop 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 shutdown_connections 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55c10c70c110 0x55c10c707da0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c10c69b2a0 0x55c10c6a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55c10c696130 0x55c10c6a3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f82697fa640 0 client.15297 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 --2- 172.21.15.23:0/2510074612 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55c10c695740 0x55c10c6a2ec0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 >> 172.21.15.23:0/2510074612 conn(0x55c10c65da60 msgr2=0x55c10c69fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 shutdown_connections 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 1 -- 172.21.15.23:0/2510074612 wait complete. 2024-03-28T15:52:24.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.469+0000 7f828aa42740 20 client.15297 trim_cache size 0 max 16384 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24494, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 112.051990577, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 92, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 116.509971857 2024-03-28T15:52:24.478 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:24.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 msgr2=0x7f826c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 0x7f826c0a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x55c10c696990 tx=0x7f8278004ab0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 msgr2=0x7f8280075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:24.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f827c00e9c0 tx=0x7f827c00ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:24.479 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 shutdown_connections 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f826c0a2800 0x7f826c0a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8280103260 0x7f8280073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f82801028a0 0x7f8280073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 --2- 172.21.15.23:0/2126973811 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f82801016b0 0x7f8280075450 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 >> 172.21.15.23:0/2126973811 conn(0x7f82800fcc00 msgr2=0x7f8280105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 shutdown_connections 2024-03-28T15:52:24.480 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:24.470+0000 7f8287e59640 1 -- 172.21.15.23:0/2126973811 wait complete. 2024-03-28T15:52:24.482 INFO:tasks.cephadm:Setup complete, yielding 2024-03-28T15:52:24.482 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T15:52:24.492 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:52:24.492 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-28T15:52:24.733 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:26.158 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:25 smithi023 ceph-mon[32448]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:27.119 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:25 smithi120 ceph-mon[36826]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:27.119 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:25 smithi117 ceph-mon[36002]: pgmap v347: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 -- 172.21.15.23:0/1046478789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 msgr2=0x7f3794103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 --2- 172.21.15.23:0/1046478789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794103840 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f37800099c0 tx=0x7f3780009cd0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 -- 172.21.15.23:0/1046478789 shutdown_connections 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 --2- 172.21.15.23:0/1046478789 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3794103d80 0x7f379410a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 --2- 172.21.15.23:0/1046478789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37941021e0 0x7f3794102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 --2- 172.21.15.23:0/1046478789 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794103840 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 -- 172.21.15.23:0/1046478789 >> 172.21.15.23:0/1046478789 conn(0x7f37940fd7a0 msgr2=0x7f37940ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:27.123 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 -- 172.21.15.23:0/1046478789 shutdown_connections 2024-03-28T15:52:27.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.545+0000 7f379ad42700 1 -- 172.21.15.23:0/1046478789 wait complete. 2024-03-28T15:52:27.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 Processor -- start 2024-03-28T15:52:27.124 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 -- start start 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 0x7f37941983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3794103d80 0x7f379419f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379410dda0 con 0x7f37941033e0 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379410dc20 con 0x7f3794103d80 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f379ad42700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379410df20 con 0x7f37941021e0 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f3793fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:27.125 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f3793fff700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:58520/0 (socket says 172.21.15.23:58520) 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f3793fff700 1 -- 172.21.15.23:0/4196480873 learned_addr learned my addr 172.21.15.23:0/4196480873 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f3798ade700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 0x7f37941983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.546+0000 7f37992df700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3794103d80 0x7f379419f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 -- 172.21.15.23:0/4196480873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 msgr2=0x7f37941983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 0x7f37941983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 -- 172.21.15.23:0/4196480873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3794103d80 msgr2=0x7f379419f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:27.126 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3794103d80 0x7f379419f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 -- 172.21.15.23:0/4196480873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37941a0020 con 0x7f37941033e0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3798ade700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 0x7f37941983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3793fff700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3780005910 tx=0x7f378000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3780009e60 con 0x7f37941033e0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3780003e40 con 0x7f37941033e0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37941a02b0 con 0x7f37941033e0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37800173f0 con 0x7f37941033e0 2024-03-28T15:52:27.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.547+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37941a0760 con 0x7f37941033e0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.548+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3794068e50 con 0x7f37941033e0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.549+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f378000fcd0 con 0x7f37941033e0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.550+0000 7f3791ffb700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 0x7f37840a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.550+0000 7f3798ade700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 0x7f37840a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.550+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f37800cfba0 con 0x7f37941033e0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.550+0000 7f3798ade700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 0x7f37840a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3794103240 tx=0x7f3788009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.553+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3780013070 con 0x7f37941033e0 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.779+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f37940625d0 con 0x7f37840a2910 2024-03-28T15:52:27.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.780+0000 7f3791ffb700 1 -- 172.21.15.23:0/4196480873 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f37940625d0 con 0x7f37840a2910 2024-03-28T15:52:27.129 INFO:teuthology.orchestra.run.smithi023.stdout:Backend: cephadm 2024-03-28T15:52:27.129 INFO:teuthology.orchestra.run.smithi023.stdout:Available: Yes 2024-03-28T15:52:27.129 INFO:teuthology.orchestra.run.smithi023.stdout:Paused: No 2024-03-28T15:52:27.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 msgr2=0x7f37840a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:27.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 0x7f37840a4dd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3794103240 tx=0x7f3788009260 comp rx=0 tx=0).stop 2024-03-28T15:52:27.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 msgr2=0x7f3794198900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f3780005910 tx=0x7f378000f690 comp rx=0 tx=0).stop 2024-03-28T15:52:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 shutdown_connections 2024-03-28T15:52:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f37840a2910 0x7f37840a4dd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3794103d80 0x7f379419f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f37941021e0 0x7f37941983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 --2- 172.21.15.23:0/4196480873 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f37941033e0 0x7f3794198900 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 >> 172.21.15.23:0/4196480873 conn(0x7f37940fd7a0 msgr2=0x7f37941066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 shutdown_connections 2024-03-28T15:52:27.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:26.783+0000 7f379ad42700 1 -- 172.21.15.23:0/4196480873 wait complete. 2024-03-28T15:52:27.589 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-28T15:52:27.834 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:28.223 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:27 smithi023 ceph-mon[32448]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:28.223 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:27 smithi023 ceph-mon[32448]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:28.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:27 smithi117 ceph-mon[36002]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:28.321 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:27 smithi117 ceph-mon[36002]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:28.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:27 smithi120 ceph-mon[36826]: pgmap v348: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:28.339 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:27 smithi120 ceph-mon[36826]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 -- 172.21.15.23:0/3329636158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc102c40 msgr2=0x7f9fdc105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3329636158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc102c40 0x7f9fdc105030 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f9fd0009990 tx=0x7f9fd0009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 -- 172.21.15.23:0/3329636158 shutdown_connections 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3329636158 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc107ea0 0x7f9fdc10a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3329636158 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc102c40 0x7f9fdc105030 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3329636158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc105570 0x7f9fdc107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 -- 172.21.15.23:0/3329636158 >> 172.21.15.23:0/3329636158 conn(0x7f9fdc0fc650 msgr2=0x7f9fdc0fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:29.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 -- 172.21.15.23:0/3329636158 shutdown_connections 2024-03-28T15:52:29.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.758+0000 7f9fe086d700 1 -- 172.21.15.23:0/3329636158 wait complete. 2024-03-28T15:52:29.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 Processor -- start 2024-03-28T15:52:29.761 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 -- start start 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 0x7f9fdc198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 0x7f9fdc198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc107ea0 0x7f9fdc19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc10dde0 con 0x7f9fdc107ea0 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc10dc60 con 0x7f9fdc102c40 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fe086d700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fdc10df60 con 0x7f9fdc105570 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fda59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 0x7f9fdc198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fd9d9b700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 0x7f9fdc198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.763 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fda59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 0x7f9fdc198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:36168/0 (socket says 172.21.15.23:36168) 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fda59c700 1 -- 172.21.15.23:0/3535110317 learned_addr learned my addr 172.21.15.23:0/3535110317 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.759+0000 7f9fdad9d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc107ea0 0x7f9fdc19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 -- 172.21.15.23:0/3535110317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 msgr2=0x7f9fdc198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 0x7f9fdc198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 -- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc107ea0 msgr2=0x7f9fdc19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc107ea0 0x7f9fdc19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 -- 172.21.15.23:0/3535110317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc1a0080 con 0x7f9fdc105570 2024-03-28T15:52:29.764 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fd9d9b700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 0x7f9fdc198960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00ea20 tx=0x7f9fcc00ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc00cbd0 con 0x7f9fdc105570 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fcc00ee60 con 0x7f9fdc105570 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fdc1a0370 con 0x7f9fdc105570 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc018680 con 0x7f9fdc105570 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.760+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fdc1a08b0 con 0x7f9fdc105570 2024-03-28T15:52:29.765 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.761+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fdc068e50 con 0x7f9fdc105570 2024-03-28T15:52:29.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.763+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9fcc018820 con 0x7f9fdc105570 2024-03-28T15:52:29.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.763+0000 7f9fcb7fe700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 0x7f9fc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:29.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.763+0000 7f9fda59c700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 0x7f9fc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:29.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.763+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9fcc0cf8d0 con 0x7f9fdc105570 2024-03-28T15:52:29.766 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.764+0000 7f9fda59c700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 0x7f9fc40a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000b2e0 tx=0x7f9fd000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:29.767 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.767+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9fcc026070 con 0x7f9fdc105570 2024-03-28T15:52:29.985 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:29.984+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fdc0625d0 con 0x7f9fc40a2890 2024-03-28T15:52:30.003 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.002+0000 7f9fcb7fe700 1 -- 172.21.15.23:0/3535110317 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4092 (secure 0 0 0) 0x7f9fdc0625d0 con 0x7f9fc40a2890 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.a smithi023 running (2m) 20s ago 2m 16.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 694630879cad 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.b smithi117 running (2m) 55s ago 2m 28.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 cbc055eb4cfa 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.c smithi120 running (2m) 55s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 07359ac86152 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.d smithi023 running (2m) 20s ago 2m 13.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 e7d17312026d 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.e smithi117 running (2m) 55s ago 2m 16.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 39baf81c7888 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.f smithi120 running (2m) 55s ago 2m 17.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 64a4d56bb72d 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.g smithi023 running (2m) 20s ago 2m 18.6M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 be382c12b7ee 2024-03-28T15:52:30.004 INFO:teuthology.orchestra.run.smithi023.stdout:mds.h smithi117 running (2m) 55s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 210c59df0a6f 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mds.i smithi120 running (2m) 55s ago 2m 19.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 d7784bfc5054 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mds.j smithi023 running (2m) 20s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 851f4e411943 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mds.k smithi117 running (2m) 55s ago 2m 16.2M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ee4a1c5f8852 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mds.l smithi120 running (2m) 55s ago 2m 15.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 7a0c590f3371 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.x smithi023 *:9283,8765 running (9m) 20s ago 9m 514M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 ebed96533914 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.y smithi117 *:8443,8765 running (7m) 55s ago 7m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f9136aa59a52 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mgr.z smithi120 *:8443,8765 running (7m) 55s ago 7m 454M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 dd81e7516a46 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mon.a smithi023 running (9m) 20s ago 9m 50.8M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 92d871696cdc 2024-03-28T15:52:30.005 INFO:teuthology.orchestra.run.smithi023.stdout:mon.b smithi117 running (8m) 55s ago 8m 43.0M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 7996064eeb22 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:mon.c smithi120 running (8m) 55s ago 8m 39.9M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 fe6917ceb952 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 smithi023 running (7m) 20s ago 7m 56.9M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 fa1e045cbcb2 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 smithi117 running (7m) 55s ago 7m 58.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 160386827f69 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 smithi120 running (6m) 55s ago 6m 47.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5650db16f7c6 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 smithi023 running (6m) 20s ago 6m 54.6M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 6d4e984c0870 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 smithi117 running (5m) 55s ago 5m 45.5M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 9db1668ece35 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 smithi120 running (5m) 55s ago 5m 57.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 126a19d823fb 2024-03-28T15:52:30.006 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 smithi023 running (5m) 20s ago 5m 41.8M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 42236009995b 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 smithi117 running (4m) 55s ago 4m 59.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 960e9b90c15d 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 smithi120 running (4m) 55s ago 4m 44.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 be0630eae8cc 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 smithi023 running (3m) 20s ago 3m 42.0M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0a5e6a3ed83d 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 smithi117 running (3m) 55s ago 3m 54.7M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 1e78d50206af 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 smithi120 running (2m) 55s ago 2m 41.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2b62c9d7592b 2024-03-28T15:52:30.007 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 msgr2=0x7f9fc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 0x7f9fc40a4d50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9fd000b2e0 tx=0x7f9fd000b880 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 msgr2=0x7f9fdc198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 0x7f9fdc198960 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00ea20 tx=0x7f9fcc00ed30 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 shutdown_connections 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9fc40a2890 0x7f9fc40a4d50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fdc107ea0 0x7f9fdc19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fdc102c40 0x7f9fdc198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 --2- 172.21.15.23:0/3535110317 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fdc105570 0x7f9fdc198960 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 >> 172.21.15.23:0/3535110317 conn(0x7f9fdc0fc650 msgr2=0x7f9fdc103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:30.008 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 shutdown_connections 2024-03-28T15:52:30.009 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:30.005+0000 7f9fe086d700 1 -- 172.21.15.23:0/3535110317 wait complete. 2024-03-28T15:52:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:30 smithi117 ceph-mon[36002]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:52:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:30 smithi120 ceph-mon[36826]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:52:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:30 smithi023 ceph-mon[32448]: pgmap v349: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-28T15:52:30.781 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-28T15:52:31.073 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:31.559 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:31 smithi023 ceph-mon[32448]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:31.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:31 smithi117 ceph-mon[36002]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:31.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:31 smithi120 ceph-mon[36826]: from='client.25102 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:32.565 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:32 smithi023 ceph-mon[32448]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:32.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:32 smithi117 ceph-mon[36002]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:32 smithi120 ceph-mon[36826]: pgmap v350: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-28T15:52:32.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.833+0000 7fc2008bf700 1 -- 172.21.15.23:0/2251828200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 msgr2=0x7fc1fc100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:32.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.833+0000 7fc2008bf700 1 --2- 172.21.15.23:0/2251828200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc100680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc1f0009990 tx=0x7fc1f0009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.835 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 -- 172.21.15.23:0/2251828200 shutdown_connections 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 --2- 172.21.15.23:0/2251828200 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc10a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 --2- 172.21.15.23:0/2251828200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 --2- 172.21.15.23:0/2251828200 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc100680 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 -- 172.21.15.23:0/2251828200 >> 172.21.15.23:0/2251828200 conn(0x7fc1fc0fbe00 msgr2=0x7fc1fc0fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 -- 172.21.15.23:0/2251828200 shutdown_connections 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.834+0000 7fc2008bf700 1 -- 172.21.15.23:0/2251828200 wait complete. 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 Processor -- start 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 -- start start 2024-03-28T15:52:32.836 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc1988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:32.838 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc10dd90 con 0x7fc1fc100220 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc10dc10 con 0x7fc1fc100bc0 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc2008bf700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1fc10df10 con 0x7fc1fc1037a0 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc1fad9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc1fad9d700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:59064/0 (socket says 172.21.15.23:59064) 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc1fad9d700 1 -- 172.21.15.23:0/816855675 learned_addr learned my addr 172.21.15.23:0/816855675 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc1fa59c700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.835+0000 7fc1f9d9b700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc1988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 -- 172.21.15.23:0/816855675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 msgr2=0x7fc1fc1988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc1988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 -- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 msgr2=0x7fc1fc198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:32.839 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 -- 172.21.15.23:0/816855675 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc19ffe0 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fa59c700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1f9d9b700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1fad9d700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400ef60 tx=0x7fc1f400ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40092e0 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1fc101330 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1f4004bb0 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4004280 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.836+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1fc101870 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.838+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1fc068e50 con 0x7fc1fc1037a0 2024-03-28T15:52:32.840 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.838+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc1f401b030 con 0x7fc1fc1037a0 2024-03-28T15:52:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.839+0000 7fc1eb7fe700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 0x7fc1e40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.839+0000 7fc1fa59c700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 0x7fc1e40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.839+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc1f40ce8b0 con 0x7fc1fc1037a0 2024-03-28T15:52:32.841 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.840+0000 7fc1fa59c700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 0x7fc1e40a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc069b60 tx=0x7fc1ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:32.846 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:32.844+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc1f4022020 con 0x7fc1fc1037a0 2024-03-28T15:52:33.084 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.084+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1fc0625d0 con 0x7fc1e40a2910 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.090+0000 7fc1eb7fe700 1 -- 172.21.15.23:0/816855675 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fc1fc0625d0 con 0x7fc1e40a2910 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stdout:mds.all 12/12 58s ago 2m smithi023=a;smithi023=d;smithi023=g;smithi023=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi120=c;smithi120=f;smithi120=i;smithi120=l;count:12 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stdout:mgr 3/3 58s ago 7m smithi023=x;smithi117=y;smithi120=z;count:3 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stdout:mon 3/3 58s ago 8m smithi023:172.21.15.23=a;smithi117:172.21.15.117=b;smithi120:172.21.15.120=c;count:3 2024-03-28T15:52:33.091 INFO:teuthology.orchestra.run.smithi023.stdout:osd 12 58s ago - 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 msgr2=0x7fc1e40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 0x7fc1e40a4dd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc069b60 tx=0x7fc1ec009260 comp rx=0 tx=0).stop 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 msgr2=0x7fc1fc19f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400ef60 tx=0x7fc1f400ef90 comp rx=0 tx=0).stop 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 shutdown_connections 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.092+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fc1e40a2910 0x7fc1e40a4dd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc1fc1037a0 0x7fc1fc19f940 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:33.093 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc1fc100220 0x7fc1fc198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:33.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 --2- 172.21.15.23:0/816855675 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1fc100bc0 0x7fc1fc1988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:33.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 >> 172.21.15.23:0/816855675 conn(0x7fc1fc0fbe00 msgr2=0x7fc1fc106840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:33.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 shutdown_connections 2024-03-28T15:52:33.094 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:33.093+0000 7fc2008bf700 1 -- 172.21.15.23:0/816855675 wait complete. 2024-03-28T15:52:33.850 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-28T15:52:34.097 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:34.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:34 smithi023 ceph-mon[32448]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:34.554 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:34 smithi023 ceph-mon[32448]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:34 smithi117 ceph-mon[36002]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:34 smithi117 ceph-mon[36002]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:34 smithi120 ceph-mon[36826]: pgmap v351: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:34 smithi120 ceph-mon[36826]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T15:52:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-28T15:52:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T15:52:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-28T15:52:35.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:35 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-28T15:52:35.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T15:52:35.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-28T15:52:35.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T15:52:35.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-28T15:52:35.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:35 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-28T15:52:35.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]: dispatch 2024-03-28T15:52:35.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-03-28T15:52:35.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-28T15:52:35.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2024-03-28T15:52:35.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:35 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]: dispatch 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 -- 172.21.15.23:0/3890026019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881022f0 msgr2=0x7f7388102750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 --2- 172.21.15.23:0/3890026019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881022f0 0x7f7388102750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f73780099c0 tx=0x7f7378009cd0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 -- 172.21.15.23:0/3890026019 shutdown_connections 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 --2- 172.21.15.23:0/3890026019 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f7388102c90 0x7f7388109540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 --2- 172.21.15.23:0/3890026019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73881010f0 0x7f7388101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 --2- 172.21.15.23:0/3890026019 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881022f0 0x7f7388102750 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 -- 172.21.15.23:0/3890026019 >> 172.21.15.23:0/3890026019 conn(0x7f73880fc6b0 msgr2=0x7f73880fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:36.011 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 -- 172.21.15.23:0/3890026019 shutdown_connections 2024-03-28T15:52:36.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.010+0000 7f738d34f700 1 -- 172.21.15.23:0/3890026019 wait complete. 2024-03-28T15:52:36.012 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 Processor -- start 2024-03-28T15:52:36.013 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 -- start start 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 0x7f7388197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 0x7f7388197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 0x7f738819e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738810ccb0 con 0x7f73881010f0 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738810cb30 con 0x7f7388102c90 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f738d34f700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738810ce30 con 0x7f73881022f0 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.011+0000 7f7386ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 0x7f7388197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73867fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 0x7f7388197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:36.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 0x7f738819e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73867fc700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 0x7f7388197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:59084/0 (socket says 172.21.15.23:59084) 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f7386ffd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 0x7f7388197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59060/0 (socket says 172.21.15.23:59060) 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73867fc700 1 -- 172.21.15.23:0/882908189 learned_addr learned my addr 172.21.15.23:0/882908189 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 -- 172.21.15.23:0/882908189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 msgr2=0x7f7388197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 0x7f7388197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 -- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 msgr2=0x7f7388197240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 0x7f7388197240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 -- 172.21.15.23:0/882908189 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738806a9d0 con 0x7f7388102c90 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f73877fe700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 0x7f738819e800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f738000efc0 tx=0x7f738000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73800092e0 con 0x7f7388102c90 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738806ac60 con 0x7f7388102c90 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7380004bb0 con 0x7f7388102c90 2024-03-28T15:52:36.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7380004280 con 0x7f7388102c90 2024-03-28T15:52:36.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.012+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738806b1a0 con 0x7f7388102c90 2024-03-28T15:52:36.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.013+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7388068e50 con 0x7f7388102c90 2024-03-28T15:52:36.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.014+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f738001b030 con 0x7f7388102c90 2024-03-28T15:52:36.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.015+0000 7f736ffff700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 0x7f73700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:36.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.015+0000 7f7386ffd700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 0x7f73700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:36.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.015+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f73800ceae0 con 0x7f7388102c90 2024-03-28T15:52:36.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.015+0000 7f7386ffd700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 0x7f73700a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7388102150 tx=0x7f737c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:36.018 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.018+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7380022020 con 0x7f7388102c90 2024-03-28T15:52:36.230 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.230+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f73880625d0 con 0x7f73700a2890 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.232+0000 7f736ffff700 1 -- 172.21.15.23:0/882908189 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f73880625d0 con 0x7f73700a2890 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stdout:HOST ADDR LABELS STATUS 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 172.21.15.23 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stdout:smithi117 172.21.15.117 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stdout:smithi120 172.21.15.120 2024-03-28T15:52:36.232 INFO:teuthology.orchestra.run.smithi023.stdout:3 hosts in cluster 2024-03-28T15:52:36.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 msgr2=0x7f73700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:36.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 0x7f73700a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7388102150 tx=0x7f737c009260 comp rx=0 tx=0).stop 2024-03-28T15:52:36.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 msgr2=0x7f738819e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:36.234 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 0x7f738819e800 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f738000efc0 tx=0x7f738000c3a0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 shutdown_connections 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f73700a2890 0x7f73700a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388102c90 0x7f738819e800 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f73881010f0 0x7f7388197240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 --2- 172.21.15.23:0/882908189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f73881022f0 0x7f7388197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 >> 172.21.15.23:0/882908189 conn(0x7f73880fc6b0 msgr2=0x7f7388105590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 shutdown_connections 2024-03-28T15:52:36.235 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:36.234+0000 7f738d34f700 1 -- 172.21.15.23:0/882908189 wait complete. 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-28T15:52:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:36 smithi117 ceph-mon[36002]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-28T15:52:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:36 smithi120 ceph-mon[36826]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: pgmap v352: 129 pgs: 129 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.0", "id": [7, 11]}]': finished 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: from='mgr.14152 172.21.15.23:0/849634232' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 8]}]': finished 2024-03-28T15:52:36.654 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:36 smithi023 ceph-mon[32448]: osdmap e93: 12 total, 12 up, 12 in 2024-03-28T15:52:37.029 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-28T15:52:37.281 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:37.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:37 smithi023 ceph-mon[32448]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:37.286 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:37 smithi023 ceph-mon[32448]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T15:52:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:37 smithi117 ceph-mon[36002]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:37 smithi117 ceph-mon[36002]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T15:52:37.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:37 smithi120 ceph-mon[36826]: from='client.25166 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:37.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:37 smithi120 ceph-mon[36826]: osdmap e94: 12 total, 12 up, 12 in 2024-03-28T15:52:38.486 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:38 smithi023 ceph-mon[32448]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-28T15:52:38.487 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:38 smithi023 ceph-mon[32448]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-28T15:52:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:38 smithi117 ceph-mon[36002]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-28T15:52:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:38 smithi117 ceph-mon[36002]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-28T15:52:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:38 smithi120 ceph-mon[36826]: pgmap v355: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-03-28T15:52:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:38 smithi120 ceph-mon[36826]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-28T15:52:39.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- 172.21.15.23:0/974581833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 msgr2=0x7f8458105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:39.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- 172.21.15.23:0/974581833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f8458105030 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8454009990 tx=0x7f8454009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- 172.21.15.23:0/974581833 shutdown_connections 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- 172.21.15.23:0/974581833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458107ea0 0x7f845810a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- 172.21.15.23:0/974581833 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f8458105030 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- 172.21.15.23:0/974581833 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458105570 0x7f8458107960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- 172.21.15.23:0/974581833 >> 172.21.15.23:0/974581833 conn(0x7f84580fc650 msgr2=0x7f84580fea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- 172.21.15.23:0/974581833 shutdown_connections 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- 172.21.15.23:0/974581833 wait complete. 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 Processor -- start 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- start start 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458105570 0x7f8458198920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458107ea0 0x7f845819f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810dd90 con 0x7f8458102c40 2024-03-28T15:52:39.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810dc10 con 0x7f8458105570 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.037+0000 7f845f313700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845810df10 con 0x7f8458107ea0 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44432/0 (socket says 172.21.15.23:44432) 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 -- 172.21.15.23:0/1278421149 learned_addr learned my addr 172.21.15.23:0/1278421149 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845c8ae700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458105570 0x7f8458198920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d8b0700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458107ea0 0x7f845819f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 -- 172.21.15.23:0/1278421149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458107ea0 msgr2=0x7f845819f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458107ea0 0x7f845819f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 -- 172.21.15.23:0/1278421149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458105570 msgr2=0x7f8458198920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458105570 0x7f8458198920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 -- 172.21.15.23:0/1278421149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84581a0040 con 0x7f8458102c40 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845d0af700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f845400bbb0 tx=0x7f845400bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:39.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454026070 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84581a02d0 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f845400f460 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8454021650 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.038+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84581a0780 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.039+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8458068e50 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.041+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8454029070 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.041+0000 7f844e7fc700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 0x7f84440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.041+0000 7f845c8ae700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 0x7f84440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.042+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f84540cfc30 con 0x7f8458102c40 2024-03-28T15:52:39.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.042+0000 7f845c8ae700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 0x7f84440a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8448009c60 tx=0x7f8448009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:39.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.046+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f845405d740 con 0x7f8458102c40 2024-03-28T15:52:39.260 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.260+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f84580625d0 con 0x7f84440a2890 2024-03-28T15:52:39.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.261+0000 7f844e7fc700 1 -- 172.21.15.23:0/1278421149 <== mgr.14152 v2:172.21.15.23:6800/1340727098 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f84580625d0 con 0x7f84440a2890 2024-03-28T15:52:39.262 INFO:teuthology.orchestra.run.smithi023.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-28T15:52:39.262 INFO:teuthology.orchestra.run.smithi023.stdout:smithi023 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103B14P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-03-28T15:52:39.262 INFO:teuthology.orchestra.run.smithi023.stdout:smithi117 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V6400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-28T15:52:39.262 INFO:teuthology.orchestra.run.smithi023.stdout:smithi120 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.263+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 msgr2=0x7f84440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.263+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 0x7f84440a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8448009c60 tx=0x7f8448009260 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 msgr2=0x7f84581983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f845400bbb0 tx=0x7f845400bbe0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 shutdown_connections 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f84440a2890 0x7f84440a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8458107ea0 0x7f845819f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8458102c40 0x7f84581983e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 --2- 172.21.15.23:0/1278421149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8458105570 0x7f8458198920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:39.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 >> 172.21.15.23:0/1278421149 conn(0x7f84580fc650 msgr2=0x7f8458103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 shutdown_connections 2024-03-28T15:52:39.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:39.264+0000 7f845f313700 1 -- 172.21.15.23:0/1278421149 wait complete. 2024-03-28T15:52:39.917 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-28T15:52:39.929 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:52:39.929 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-28T15:52:40.174 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:40 smithi117 ceph-mon[36002]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 290 B/s, 0 objects/s recovering 2024-03-28T15:52:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:40 smithi117 ceph-mon[36002]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:40 smithi120 ceph-mon[36826]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 290 B/s, 0 objects/s recovering 2024-03-28T15:52:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:40 smithi120 ceph-mon[36826]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:40 smithi023 ceph-mon[32448]: pgmap v356: 129 pgs: 2 peering, 127 active+clean; 641 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 290 B/s, 0 objects/s recovering 2024-03-28T15:52:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:40 smithi023 ceph-mon[32448]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:52:41.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:41 smithi023 ceph-mon[32448]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-28T15:52:41.599 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:41 smithi023 ceph-mon[32448]: Cluster is now healthy 2024-03-28T15:52:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:41 smithi117 ceph-mon[36002]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-28T15:52:41.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:41 smithi117 ceph-mon[36002]: Cluster is now healthy 2024-03-28T15:52:41.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:41 smithi120 ceph-mon[36826]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-28T15:52:41.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:41 smithi120 ceph-mon[36826]: Cluster is now healthy 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 -- 172.21.15.23:0/3335341287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc1010c0 msgr2=0x7f55fc1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 --2- 172.21.15.23:0/3335341287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc1010c0 0x7f55fc1014e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f55f00098f0 tx=0x7f55f0009c00 comp rx=0 tx=0).stop 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 -- 172.21.15.23:0/3335341287 shutdown_connections 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 --2- 172.21.15.23:0/3335341287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc102c60 0x7f55fc109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 --2- 172.21.15.23:0/3335341287 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc1010c0 0x7f55fc1014e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 --2- 172.21.15.23:0/3335341287 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1022c0 0x7f55fc102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 -- 172.21.15.23:0/3335341287 >> 172.21.15.23:0/3335341287 conn(0x7f55fc0fc660 msgr2=0x7f55fc0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 -- 172.21.15.23:0/3335341287 shutdown_connections 2024-03-28T15:52:41.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.853+0000 7f5602c29700 1 -- 172.21.15.23:0/3335341287 wait complete. 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 Processor -- start 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 -- start start 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc1022c0 0x7f55fc198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc102c60 0x7f55fc19fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc10ccb0 con 0x7f55fc1010c0 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc10cb30 con 0x7f55fc1022c0 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f5602c29700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55fc10ce30 con 0x7f55fc102c60 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f56009c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:41.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.854+0000 7f56009c5700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44452/0 (socket says 172.21.15.23:44452) 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 -- 172.21.15.23:0/3135222954 learned_addr learned my addr 172.21.15.23:0/3135222954 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f55fbfff700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc1022c0 0x7f55fc198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56011c6700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc102c60 0x7f55fc19fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 -- 172.21.15.23:0/3135222954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc102c60 msgr2=0x7f55fc19fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc102c60 0x7f55fc19fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 -- 172.21.15.23:0/3135222954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc1022c0 msgr2=0x7f55fc198a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc1022c0 0x7f55fc198a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 -- 172.21.15.23:0/3135222954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc1a0130 con 0x7f55fc1010c0 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f56009c5700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009230 tx=0x7f55f00050c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:41.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0005440 con 0x7f55fc1010c0 2024-03-28T15:52:41.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f55f00055e0 con 0x7f55fc1010c0 2024-03-28T15:52:41.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55fc1034e0 con 0x7f55fc1010c0 2024-03-28T15:52:41.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0003e40 con 0x7f55fc1010c0 2024-03-28T15:52:41.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.855+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55fc1039c0 con 0x7f55fc1010c0 2024-03-28T15:52:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.856+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55fc068e50 con 0x7f55fc1010c0 2024-03-28T15:52:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.858+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f55f000b560 con 0x7f55fc1010c0 2024-03-28T15:52:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.859+0000 7f55f9ffb700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 0x7f55ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.859+0000 7f55fbfff700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 0x7f55ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:41.860 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.859+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f55f00cfde0 con 0x7f55fc1010c0 2024-03-28T15:52:41.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.859+0000 7f55fbfff700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 0x7f55ec0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009750 tx=0x7f55e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:41.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:41.863+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f55f0013030 con 0x7f55fc1010c0 2024-03-28T15:52:42.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.115+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f55fc103e50 con 0x7f55fc1010c0 2024-03-28T15:52:42.115 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.115+0000 7f55f9ffb700 1 -- 172.21.15.23:0/3135222954 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 28 v28) v1 ==== 76+0+3038 (secure 0 0 0) 0x7f55f009cb20 con 0x7f55fc1010c0 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:e28 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:legacy client fscid: 1 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:Filesystem 'cephfs' (1) 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:fs_name cephfs 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 28 2024-03-28T15:52:42.116 INFO:teuthology.orchestra.run.smithi023.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-28T15:50:31.785133+0000 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-28T15:51:00.222875+0000 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:tableserver 0 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:root 0 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:session_timeout 300 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:session_autoclose 300 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:max_file_size 1099511627776 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:max_xattr_size 65536 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:required_client_features {} 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure 0 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:last_failure_osd_epoch 0 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:max_mds 5 2024-03-28T15:52:42.117 INFO:teuthology.orchestra.run.smithi023.stdout:in 0,1,2,3,4 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:up {0=24494,1=24481,2=24455,3=24454,4=14640} 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:failed 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:damaged 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:stopped 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:data_pools [3,4] 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:metadata_pool 2 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:inline_data disabled 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:balancer 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:bal_rank_mask -1 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:standby_count_wanted 0 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:qdb_cluster leader: 24494 members: 24481,24494,24455,24454,14640 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.b{0:24494} state up:active seq 3 addr [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.118 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.f{0:24442} state up:standby-replay seq 1 addr [v2:172.21.15.120:6834/1615812668,v1:172.21.15.120:6835/1615812668] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.i{1:24481} state up:active seq 11 addr [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.e{1:24422} state up:standby-replay seq 1 addr [v2:172.21.15.117:6834/3656482331,v1:172.21.15.117:6835/3656482331] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.h{2:24455} state up:active seq 12 addr [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.l{2:24403} state up:standby-replay seq 1 addr [v2:172.21.15.120:6832/515874083,v1:172.21.15.120:6833/515874083] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.c{3:24454} state up:active seq 13 addr [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.k{3:24392} state up:standby-replay seq 1 addr [v2:172.21.15.117:6832/2721990059,v1:172.21.15.117:6833/2721990059] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.a{4:14640} state up:active seq 10 addr [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.g{4:14628} state up:standby-replay seq 1 addr [v2:172.21.15.23:6838/1641251810,v1:172.21.15.23:6839/1641251810] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.119 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:42.120 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:42.120 INFO:teuthology.orchestra.run.smithi023.stdout:Standby daemons: 2024-03-28T15:52:42.120 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:42.120 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.j{-1:14553} state up:standby seq 1 addr [v2:172.21.15.23:6834/1890623637,v1:172.21.15.23:6835/1890623637] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.120 INFO:teuthology.orchestra.run.smithi023.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.23:6836/252463346,v1:172.21.15.23:6837/252463346] compat {c=[1],r=[1],i=[fff]}] 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 msgr2=0x7f55ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 0x7f55ec0a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f55e8009750 tx=0x7f55e8009310 comp rx=0 tx=0).stop 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 msgr2=0x7f55fc198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009230 tx=0x7f55f00050c0 comp rx=0 tx=0).stop 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 shutdown_connections 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f55ec0a2890 0x7f55ec0a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f55fc102c60 0x7f55fc19fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f55fc1010c0 0x7f55fc198520 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:42.121 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 --2- 172.21.15.23:0/3135222954 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55fc1022c0 0x7f55fc198a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 >> 172.21.15.23:0/3135222954 conn(0x7f55fc0fc660 msgr2=0x7f55fc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 shutdown_connections 2024-03-28T15:52:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:42.118+0000 7f5602c29700 1 -- 172.21.15.23:0/3135222954 wait complete. 2024-03-28T15:52:42.122 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:42.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:42 smithi023 ceph-mon[32448]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:42.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:42 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3135222954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T15:52:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:42 smithi117 ceph-mon[36002]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:42 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3135222954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T15:52:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:42 smithi120 ceph-mon[36826]: pgmap v357: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:42 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3135222954' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-28T15:52:42.846 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-28T15:52:43.127 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:44.618 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:44 smithi023 ceph-mon[32448]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:44 smithi117 ceph-mon[36002]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:44 smithi120 ceph-mon[36826]: pgmap v358: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 304 B/s, 1 objects/s recovering 2024-03-28T15:52:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 -- 172.21.15.23:0/432763005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838074b70 msgr2=0x7f9838074ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:44.853 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 --2- 172.21.15.23:0/432763005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838074b70 0x7f9838074ff0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9828009990 tx=0x7f9828009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 -- 172.21.15.23:0/432763005 shutdown_connections 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 --2- 172.21.15.23:0/432763005 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f983810a0e0 0x7f983810c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 --2- 172.21.15.23:0/432763005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838076140 0x7f98380745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 --2- 172.21.15.23:0/432763005 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838074b70 0x7f9838074ff0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 -- 172.21.15.23:0/432763005 >> 172.21.15.23:0/432763005 conn(0x7f98380fd7a0 msgr2=0x7f98380ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 -- 172.21.15.23:0/432763005 shutdown_connections 2024-03-28T15:52:44.854 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.853+0000 7f983f3bd700 1 -- 172.21.15.23:0/432763005 wait complete. 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 Processor -- start 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 -- start start 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 0x7f98381a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983810ffb0 con 0x7f9838076140 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983810fe30 con 0x7f983810a0e0 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983f3bd700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9838110130 con 0x7f9838074b70 2024-03-28T15:52:44.856 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983c958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983d159700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983c958700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44472/0 (socket says 172.21.15.23:44472) 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983d159700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:59540/0 (socket says 172.21.15.23:59540) 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983c958700 1 -- 172.21.15.23:0/630251753 learned_addr learned my addr 172.21.15.23:0/630251753 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.854+0000 7f983d95a700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 0x7f98381a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 -- 172.21.15.23:0/630251753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 msgr2=0x7f98381a1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 0x7f98381a1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 -- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 msgr2=0x7f983819aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 -- 172.21.15.23:0/630251753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98381a21f0 con 0x7f9838074b70 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983c958700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d95a700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 0x7f98381a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983d159700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f983400ea20 tx=0x7f983400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:44.857 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983400cbd0 con 0x7f9838074b70 2024-03-28T15:52:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98381a24e0 con 0x7f9838074b70 2024-03-28T15:52:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f983400ee60 con 0x7f9838074b70 2024-03-28T15:52:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9834018680 con 0x7f9838074b70 2024-03-28T15:52:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.855+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98381a2a20 con 0x7f9838074b70 2024-03-28T15:52:44.858 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.856+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9838068e50 con 0x7f9838074b70 2024-03-28T15:52:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.857+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9834047a60 con 0x7f9838074b70 2024-03-28T15:52:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.858+0000 7f982e7fc700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 0x7f98240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.858+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f98340cefa0 con 0x7f9838074b70 2024-03-28T15:52:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.858+0000 7f983c958700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 0x7f98240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:44.859 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.858+0000 7f983c958700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 0x7f98240a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f982800b2e0 tx=0x7f982800b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:44.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:44.861+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f983409bbf0 con 0x7f9838074b70 2024-03-28T15:52:45.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.068+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f9838062da0 con 0x7f9838074b70 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.069+0000 7f982e7fc700 1 -- 172.21.15.23:0/630251753 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v94) v1 ==== 56+0+5739 (secure 0 0 0) 0x7f983409b4f0 con 0x7f9838074b70 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stdout:epoch 94 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stdout:fsid a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stdout:created 2024-03-28T15:42:38.302272+0000 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stdout:modified 2024-03-28T15:52:36.259332+0000 2024-03-28T15:52:45.069 INFO:teuthology.orchestra.run.smithi023.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:crush_version 27 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:full_ratio 0.95 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:backfillfull_ratio 0.9 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:nearfull_ratio 0.85 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:require_min_compat_client luminous 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:min_compat_client luminous 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:require_osd_release squid 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:stretch_mode_enabled false 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 90 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:max_osd 12 2024-03-28T15:52:45.070 INFO:teuthology.orchestra.run.smithi023.stdout:osd.0 up in weight 1 up_from 8 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6802/3911626101,v1:172.21.15.23:6803/3911626101] [v2:172.21.15.23:6804/3911626101,v1:172.21.15.23:6805/3911626101] exists,up d73a4797-07a2-4f5c-9df2-ae9b5ec8c617 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.1 up in weight 1 up_from 13 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/1603075962,v1:172.21.15.117:6801/1603075962] [v2:172.21.15.117:6802/1603075962,v1:172.21.15.117:6803/1603075962] exists,up 6fe2a5d7-6406-4527-bf36-1ad15e345a54 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.2 up in weight 1 up_from 18 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6800/399817825,v1:172.21.15.120:6801/399817825] [v2:172.21.15.120:6802/399817825,v1:172.21.15.120:6803/399817825] exists,up 9c0639e5-8825-479c-a148-f3a525d6995c 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.3 up in weight 1 up_from 26 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6810/4165771629,v1:172.21.15.23:6811/4165771629] [v2:172.21.15.23:6812/4165771629,v1:172.21.15.23:6813/4165771629] exists,up a5757ad6-b03a-4985-ba33-e0010a2c3161 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.4 up in weight 1 up_from 34 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/1702661707,v1:172.21.15.117:6809/1702661707] [v2:172.21.15.117:6810/1702661707,v1:172.21.15.117:6811/1702661707] exists,up 884d0555-eb5d-4206-8931-26bf05fe7e40 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.5 up in weight 1 up_from 39 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6808/3042970364,v1:172.21.15.120:6809/3042970364] [v2:172.21.15.120:6810/3042970364,v1:172.21.15.120:6811/3042970364] exists,up 0743e053-1af5-44cd-9d42-efd7079b15f4 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.6 up in weight 1 up_from 47 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6818/2799436547,v1:172.21.15.23:6819/2799436547] [v2:172.21.15.23:6820/2799436547,v1:172.21.15.23:6821/2799436547] exists,up 21218a07-2898-42e9-a82b-935c3bb50fcb 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.7 up in weight 1 up_from 52 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/3265640789,v1:172.21.15.117:6817/3265640789] [v2:172.21.15.117:6818/3265640789,v1:172.21.15.117:6819/3265640789] exists,up 18720d2e-c5ab-4b3e-b8ee-8b2b687daabf 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.8 up in weight 1 up_from 60 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6816/2425649458,v1:172.21.15.120:6817/2425649458] [v2:172.21.15.120:6818/2425649458,v1:172.21.15.120:6819/2425649458] exists,up 692e7ca1-8a89-4b1d-a931-f7e9bb788110 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.9 up in weight 1 up_from 68 up_thru 91 down_at 0 last_clean_interval [0,0) [v2:172.21.15.23:6826/3759918369,v1:172.21.15.23:6827/3759918369] [v2:172.21.15.23:6828/3759918369,v1:172.21.15.23:6829/3759918369] exists,up 30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/2718715986,v1:172.21.15.117:6825/2718715986] [v2:172.21.15.117:6826/2718715986,v1:172.21.15.117:6827/2718715986] exists,up 52537bc3-64df-47b7-85c8-6b32eddd8296 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:osd.11 up in weight 1 up_from 81 up_thru 93 down_at 0 last_clean_interval [0,0) [v2:172.21.15.120:6824/3052698100,v1:172.21.15.120:6825/3052698100] [v2:172.21.15.120:6826/3052698100,v1:172.21.15.120:6827/3052698100] exists,up d7c32944-dd01-42b8-9b33-5aed77590035 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.0 [7,11] 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.c [2,4] 2024-03-28T15:52:45.071 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.10 [8,11] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 2.1b [2,1] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.c [3,9] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.e [4,8] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.11 [4,6] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.17 [4,8] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.18 [3,9] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.21 [5,9] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 3.31 [4,7] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.b [9,8] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.c [9,10] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.e [9,11] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:pg_upmap_items 4.1d [5,8] 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2505463471 expires 2024-03-29T15:43:34.285406+0000 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/3395482026 expires 2024-03-29T15:43:34.285406+0000 2024-03-28T15:52:45.072 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/4208679988 expires 2024-03-29T15:43:34.285406+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/4208679988 expires 2024-03-29T15:43:34.285406+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6800/3484701507 expires 2024-03-29T15:43:06.191436+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/2095255310 expires 2024-03-29T15:43:06.191436+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:6801/3484701507 expires 2024-03-29T15:43:06.191436+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1534223304 expires 2024-03-29T15:43:06.191436+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/611179097 expires 2024-03-29T15:43:06.191436+0000 2024-03-28T15:52:45.073 INFO:teuthology.orchestra.run.smithi023.stdout:blocklist 172.21.15.23:0/1777550602 expires 2024-03-29T15:43:34.285406+0000 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 msgr2=0x7f98240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 0x7f98240a4d50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f982800b2e0 tx=0x7f982800b880 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 msgr2=0x7f983819a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f983400ea20 tx=0x7f983400ed30 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 shutdown_connections 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f98240a2890 0x7f98240a4d50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f983810a0e0 0x7f98381a1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9838074b70 0x7f983819a590 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 --2- 172.21.15.23:0/630251753 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9838076140 0x7f983819aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 >> 172.21.15.23:0/630251753 conn(0x7f98380fd7a0 msgr2=0x7f9838106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 shutdown_connections 2024-03-28T15:52:45.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.071+0000 7f983f3bd700 1 -- 172.21.15.23:0/630251753 wait complete. 2024-03-28T15:52:45.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:45 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/630251753' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T15:52:45.745 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-28T15:52:45.757 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:45.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:45 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/630251753' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T15:52:45.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:45 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/630251753' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 -- 172.21.15.23:0/3940163507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20104b50 msgr2=0x7f6a20106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 --2- 172.21.15.23:0/3940163507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20104b50 0x7f6a20106f40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6a14009900 tx=0x7f6a1402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 -- 172.21.15.23:0/3940163507 shutdown_connections 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 --2- 172.21.15.23:0/3940163507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 0x7f6a20109870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 --2- 172.21.15.23:0/3940163507 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20104b50 0x7f6a20106f40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 --2- 172.21.15.23:0/3940163507 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20102220 0x7f6a20104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 -- 172.21.15.23:0/3940163507 >> 172.21.15.23:0/3940163507 conn(0x7f6a200fbbd0 msgr2=0x7f6a200fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 -- 172.21.15.23:0/3940163507 shutdown_connections 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.863+0000 7f6a26019640 1 -- 172.21.15.23:0/3940163507 wait complete. 2024-03-28T15:52:45.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 Processor -- start 2024-03-28T15:52:45.865 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 -- start start 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20104b50 0x7f6a201001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 0x7f6a20100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010d370 con 0x7f6a20102220 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010d1f0 con 0x7f6a20107480 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a26019640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010d4f0 con 0x7f6a20104b50 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a1f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a1f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44500/0 (socket says 172.21.15.23:44500) 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a1f7fe640 1 -- 172.21.15.23:0/2545682462 learned_addr learned my addr 172.21.15.23:0/2545682462 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a1effd640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20104b50 0x7f6a201001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.864+0000 7f6a1ffff640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 0x7f6a20100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 -- 172.21.15.23:0/2545682462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20104b50 msgr2=0x7f6a201001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:45.867 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20104b50 0x7f6a201001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 -- 172.21.15.23:0/2545682462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 msgr2=0x7f6a20100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 0x7f6a20100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 -- 172.21.15.23:0/2545682462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a20100fa0 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1f7fe640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000d950 tx=0x7f6a1000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a10017030 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a201a1b30 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a10004020 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a100047a0 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.865+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a201a20a0 con 0x7f6a20102220 2024-03-28T15:52:45.868 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.866+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a200fd6c0 con 0x7f6a20102220 2024-03-28T15:52:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.867+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6a100041d0 con 0x7f6a20102220 2024-03-28T15:52:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.868+0000 7f6a1cff9640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.868+0000 7f6a1effd640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.868+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f6a100ce890 con 0x7f6a20102220 2024-03-28T15:52:45.869 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.868+0000 7f6a1effd640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 0x7f6a080a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6a1402c170 tx=0x7f6a1402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:45.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:45.873+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6a1009b540 con 0x7f6a20102220 2024-03-28T15:52:46.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.140+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a20062340 con 0x7f6a20102220 2024-03-28T15:52:46.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.142+0000 7f6a1cff9640 1 -- 172.21.15.23:0/2545682462 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24913 (secure 0 0 0) 0x7f6a1009ae40 con 0x7f6a20102220 2024-03-28T15:52:46.143 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:46.143 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":94,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:52:36.259332+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 msgr2=0x7f6a080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 0x7f6a080a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6a1402c170 tx=0x7f6a1402df90 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 msgr2=0x7f6a20101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a1000d950 tx=0x7f6a1000de10 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 shutdown_connections 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6a080a2800 0x7f6a080a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a20107480 0x7f6a20100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6a20104b50 0x7f6a201001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 --2- 172.21.15.23:0/2545682462 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6a20102220 0x7f6a20101b90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.144+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 >> 172.21.15.23:0/2545682462 conn(0x7f6a200fbbd0 msgr2=0x7f6a20102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.145 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.145+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 shutdown_connections 2024-03-28T15:52:46.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.145+0000 7f6a26019640 1 -- 172.21.15.23:0/2545682462 wait complete. 2024-03-28T15:52:46.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:52:46.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 -- 172.21.15.23:0/349402985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 msgr2=0x7f42481009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 --2- 172.21.15.23:0/349402985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f42481009c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f4238009900 tx=0x7f423802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 -- 172.21.15.23:0/349402985 shutdown_connections 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 --2- 172.21.15.23:0/349402985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248102150 0x7f42481089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 --2- 172.21.15.23:0/349402985 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248101790 0x7f4248101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 --2- 172.21.15.23:0/349402985 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f42481009c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 -- 172.21.15.23:0/349402985 >> 172.21.15.23:0/349402985 conn(0x7f42480fbb30 msgr2=0x7f42480fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 -- 172.21.15.23:0/349402985 shutdown_connections 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 -- 172.21.15.23:0/349402985 wait complete. 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.263+0000 7f424e516640 1 Processor -- start 2024-03-28T15:52:46.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 -- start start 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 0x7f4248197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248102150 0x7f424819f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424810c230 con 0x7f42481005a0 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424810c0b0 con 0x7f4248101790 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424e516640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424810c3b0 con 0x7f4248102150 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44516/0 (socket says 172.21.15.23:44516) 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 -- 172.21.15.23:0/2794361647 learned_addr learned my addr 172.21.15.23:0/2794361647 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f42477fe640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 0x7f4248197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f424cab3640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248102150 0x7f424819f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 -- 172.21.15.23:0/2794361647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248102150 msgr2=0x7f424819f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248102150 0x7f424819f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 -- 172.21.15.23:0/2794361647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 msgr2=0x7f4248197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 0x7f4248197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f4247fff640 1 -- 172.21.15.23:0/2794361647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424819f680 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.264+0000 7f42477fe640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 0x7f4248197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f4247fff640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4238000c00 tx=0x7f423802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238041070 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4238004450 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f424819f910 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238003df0 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.265+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424819fe00 con 0x7f42481005a0 2024-03-28T15:52:46.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.266+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424819ffd0 con 0x7f42481005a0 2024-03-28T15:52:46.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.267+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4238044070 con 0x7f42481005a0 2024-03-28T15:52:46.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.268+0000 7f42457fa640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 0x7f42300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.268+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f42380f0b10 con 0x7f42481005a0 2024-03-28T15:52:46.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.268+0000 7f42477fe640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 0x7f42300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.268+0000 7f42477fe640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 0x7f42300a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4248198f70 tx=0x7f423c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.273 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.273+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f42380bd850 con 0x7f42481005a0 2024-03-28T15:52:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:46 smithi023 ceph-mon[32448]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 266 B/s, 1 objects/s recovering 2024-03-28T15:52:46.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:46 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2545682462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:46.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.460+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f42481a0360 con 0x7f42481005a0 2024-03-28T15:52:46.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.460+0000 7f42457fa640 1 -- 172.21.15.23:0/2794361647 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f42380bd150 con 0x7f42481005a0 2024-03-28T15:52:46.460 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 msgr2=0x7f42300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 0x7f42300a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f4248198f70 tx=0x7f423c005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 msgr2=0x7f4248197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4238000c00 tx=0x7f423802c640 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 shutdown_connections 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f42300a2800 0x7f42300a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4248102150 0x7f424819f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4248101790 0x7f4248197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 --2- 172.21.15.23:0/2794361647 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42481005a0 0x7f4248197a60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 >> 172.21.15.23:0/2794361647 conn(0x7f42480fbb30 msgr2=0x7f4248068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 shutdown_connections 2024-03-28T15:52:46.463 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.462+0000 7f424e516640 1 -- 172.21.15.23:0/2794361647 wait complete. 2024-03-28T15:52:46.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:52:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 -- 172.21.15.23:0/311251847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b581005a0 msgr2=0x7f6b581009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 --2- 172.21.15.23:0/311251847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b581005a0 0x7f6b581009c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f6b48009900 tx=0x7f6b4802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 -- 172.21.15.23:0/311251847 shutdown_connections 2024-03-28T15:52:46.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 --2- 172.21.15.23:0/311251847 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b581089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 --2- 172.21.15.23:0/311251847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b58101790 0x7f6b58101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 --2- 172.21.15.23:0/311251847 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b581005a0 0x7f6b581009c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 -- 172.21.15.23:0/311251847 >> 172.21.15.23:0/311251847 conn(0x7f6b580fbb30 msgr2=0x7f6b580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 -- 172.21.15.23:0/311251847 shutdown_connections 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.585+0000 7f6b5de27640 1 -- 172.21.15.23:0/311251847 wait complete. 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 Processor -- start 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 -- start start 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 0x7f6b58197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b5819efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5810c260 con 0x7f6b58101790 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5810c0e0 con 0x7f6b581005a0 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b5de27640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5810c3e0 con 0x7f6b58102150 2024-03-28T15:52:46.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b56ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b57fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b5819efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b56ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44534/0 (socket says 172.21.15.23:44534) 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b56ffd640 1 -- 172.21.15.23:0/3345263443 learned_addr learned my addr 172.21.15.23:0/3345263443 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.586+0000 7f6b577fe640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 0x7f6b58197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 -- 172.21.15.23:0/3345263443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 msgr2=0x7f6b5819efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b5819efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 -- 172.21.15.23:0/3345263443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 msgr2=0x7f6b58197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 0x7f6b58197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 -- 172.21.15.23:0/3345263443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5819f650 con 0x7f6b58101790 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b57fff640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b5819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b577fe640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 0x7f6b58197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b56ffd640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00ea60 tx=0x7f6b4c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c00cde0 con 0x7f6b58101790 2024-03-28T15:52:46.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5819f940 con 0x7f6b58101790 2024-03-28T15:52:46.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b4c0035f0 con 0x7f6b58101790 2024-03-28T15:52:46.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c010670 con 0x7f6b58101790 2024-03-28T15:52:46.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.587+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5819fee0 con 0x7f6b58101790 2024-03-28T15:52:46.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.588+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b581a0170 con 0x7f6b58101790 2024-03-28T15:52:46.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.589+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6b4c0037a0 con 0x7f6b58101790 2024-03-28T15:52:46.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.590+0000 7f6b54ff9640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 0x7f6b400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.590+0000 7f6b577fe640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 0x7f6b400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.590+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f6b4c0cf200 con 0x7f6b58101790 2024-03-28T15:52:46.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.590+0000 7f6b577fe640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 0x7f6b400a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6b4802c170 tx=0x7f6b4802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.595+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6b4c09bf40 con 0x7f6b58101790 2024-03-28T15:52:46.779 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.779+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b581a0340 con 0x7f6b58101790 2024-03-28T15:52:46.780 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.780+0000 7f6b54ff9640 1 -- 172.21.15.23:0/3345263443 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b4c09b840 con 0x7f6b58101790 2024-03-28T15:52:46.780 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 msgr2=0x7f6b400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 0x7f6b400a4cc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6b4802c170 tx=0x7f6b4802df90 comp rx=0 tx=0).stop 2024-03-28T15:52:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 msgr2=0x7f6b58197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.782 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00ea60 tx=0x7f6b4c00ef20 comp rx=0 tx=0).stop 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 shutdown_connections 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f6b400a2800 0x7f6b400a4cc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6b58102150 0x7f6b5819efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6b58101790 0x7f6b58197f70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 --2- 172.21.15.23:0/3345263443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b581005a0 0x7f6b58197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 >> 172.21.15.23:0/3345263443 conn(0x7f6b580fbb30 msgr2=0x7f6b580fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 shutdown_connections 2024-03-28T15:52:46.783 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.782+0000 7f6b5de27640 1 -- 172.21.15.23:0/3345263443 wait complete. 2024-03-28T15:52:46.793 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:52:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:46 smithi117 ceph-mon[36002]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 266 B/s, 1 objects/s recovering 2024-03-28T15:52:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:46 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2545682462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:46 smithi120 ceph-mon[36826]: pgmap v359: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 266 B/s, 1 objects/s recovering 2024-03-28T15:52:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:46 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2545682462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- 172.21.15.23:0/2522414014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd358101790 msgr2=0x7fd358101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 --2- 172.21.15.23:0/2522414014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd358101790 0x7fd358101c10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fd348009900 tx=0x7fd34802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- 172.21.15.23:0/2522414014 shutdown_connections 2024-03-28T15:52:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 --2- 172.21.15.23:0/2522414014 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 0x7fd3581089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 --2- 172.21.15.23:0/2522414014 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd358101790 0x7fd358101c10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 --2- 172.21.15.23:0/2522414014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3581005a0 0x7fd3581009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- 172.21.15.23:0/2522414014 >> 172.21.15.23:0/2522414014 conn(0x7fd3580fbb30 msgr2=0x7fd3580fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- 172.21.15.23:0/2522414014 shutdown_connections 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- 172.21.15.23:0/2522414014 wait complete. 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 Processor -- start 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.899+0000 7fd360663640 1 -- start start 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358101790 0x7fd35819c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 0x7fd35819ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35810c270 con 0x7fd3581005a0 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35810c0f0 con 0x7fd358101790 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35810c3f0 con 0x7fd358102150 2024-03-28T15:52:46.902 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44544/0 (socket says 172.21.15.23:44544) 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 -- 172.21.15.23:0/1840419634 learned_addr learned my addr 172.21.15.23:0/1840419634 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35dbfe640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358101790 0x7fd35819c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35ec00640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 0x7fd35819ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 -- 172.21.15.23:0/1840419634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 msgr2=0x7fd35819ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 0x7fd35819ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 -- 172.21.15.23:0/1840419634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358101790 msgr2=0x7fd35819c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358101790 0x7fd35819c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 -- 172.21.15.23:0/1840419634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35819f660 con 0x7fd3581005a0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd35e3ff640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd35400e9c0 tx=0x7fd35400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35400cde0 con 0x7fd3581005a0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.900+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35819f8f0 con 0x7fd3581005a0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.901+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd354010910 con 0x7fd3581005a0 2024-03-28T15:52:46.903 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.901+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd354018a10 con 0x7fd3581005a0 2024-03-28T15:52:46.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.901+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35819fe10 con 0x7fd3581005a0 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.901+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd358107940 con 0x7fd3581005a0 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.903+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd35401f020 con 0x7fd3581005a0 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.903+0000 7fd34f7fe640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 0x7fd3440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.903+0000 7fd35dbfe640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 0x7fd3440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.903+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fd3540cf010 con 0x7fd3581005a0 2024-03-28T15:52:46.905 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.904+0000 7fd35dbfe640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 0x7fd3440a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd34802c240 tx=0x7fd34802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:46.911 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:46.911+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd35409bcc0 con 0x7fd3581005a0 2024-03-28T15:52:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.096+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd358062340 con 0x7fd3581005a0 2024-03-28T15:52:47.096 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.096+0000 7fd34f7fe640 1 -- 172.21.15.23:0/1840419634 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd35409b5c0 con 0x7fd3581005a0 2024-03-28T15:52:47.097 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:52:47.098 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 msgr2=0x7fd3440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 0x7fd3440a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd34802c240 tx=0x7fd34802df90 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 msgr2=0x7fd358069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd35400e9c0 tx=0x7fd35400ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 shutdown_connections 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd3440a2800 0x7fd3440a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd358102150 0x7fd35819ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd358101790 0x7fd35819c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 --2- 172.21.15.23:0/1840419634 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd3581005a0 0x7fd358069370 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 >> 172.21.15.23:0/1840419634 conn(0x7fd3580fbb30 msgr2=0x7fd358104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 shutdown_connections 2024-03-28T15:52:47.099 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.098+0000 7fd360663640 1 -- 172.21.15.23:0/1840419634 wait complete. 2024-03-28T15:52:47.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:52:47.217 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 -- 172.21.15.23:0/34224977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10441028a0 msgr2=0x7f1044102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 --2- 172.21.15.23:0/34224977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10441028a0 0x7f1044102d20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1038009900 tx=0x7f103802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 -- 172.21.15.23:0/34224977 shutdown_connections 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 --2- 172.21.15.23:0/34224977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1044103260 0x7f1044109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 --2- 172.21.15.23:0/34224977 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f10441028a0 0x7f1044102d20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 --2- 172.21.15.23:0/34224977 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441016b0 0x7f1044101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 -- 172.21.15.23:0/34224977 >> 172.21.15.23:0/34224977 conn(0x7f10440fcc00 msgr2=0x7f10440ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 -- 172.21.15.23:0/34224977 shutdown_connections 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.215+0000 7f1049c81640 1 -- 172.21.15.23:0/34224977 wait complete. 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 Processor -- start 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 -- start start 2024-03-28T15:52:47.218 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10441016b0 0x7f1044075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441028a0 0x7f1044073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104410d380 con 0x7f1044103260 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104410d200 con 0x7f10441016b0 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1049c81640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104410d500 con 0x7f10441028a0 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1043fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1043fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44554/0 (socket says 172.21.15.23:44554) 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1043fff640 1 -- 172.21.15.23:0/2710107099 learned_addr learned my addr 172.21.15.23:0/2710107099 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f10437fe640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10441016b0 0x7f1044075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.216+0000 7f1042ffd640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441028a0 0x7f1044073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 -- 172.21.15.23:0/2710107099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441028a0 msgr2=0x7f1044073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.219 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441028a0 0x7f1044073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 -- 172.21.15.23:0/2710107099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10441016b0 msgr2=0x7f1044075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10441016b0 0x7f1044075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 -- 172.21.15.23:0/2710107099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044074860 con 0x7f1044103260 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1043fff640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f103c00eb10 tx=0x7f103c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c004b20 con 0x7f1044103260 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f103c004cc0 con 0x7f1044103260 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10441a1b20 con 0x7f1044103260 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c010760 con 0x7f1044103260 2024-03-28T15:52:47.220 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.217+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10441a1ff0 con 0x7f1044103260 2024-03-28T15:52:47.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.218+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1044067c00 con 0x7f1044103260 2024-03-28T15:52:47.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.219+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f103c005020 con 0x7f1044103260 2024-03-28T15:52:47.221 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.220+0000 7f1040ff9640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 0x7f102c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.220+0000 7f10437fe640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 0x7f102c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.220+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f103c0cf390 con 0x7f1044103260 2024-03-28T15:52:47.222 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.220+0000 7f10437fe640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 0x7f102c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1044102700 tx=0x7f1034005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.224 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.224+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f103c09c0e0 con 0x7f1044103260 2024-03-28T15:52:47.467 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.467+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f104402d0f0 con 0x7f1044103260 2024-03-28T15:52:47.468 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.468+0000 7f1040ff9640 1 -- 172.21.15.23:0/2710107099 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7f103c09b9e0 con 0x7f1044103260 2024-03-28T15:52:47.468 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 msgr2=0x7f102c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 0x7f102c0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1044102700 tx=0x7f1034005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 msgr2=0x7f1044073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f103c00eb10 tx=0x7f103c00c2d0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.470 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 shutdown_connections 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f102c0a2800 0x7f102c0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1044103260 0x7f1044073fd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f10441028a0 0x7f1044073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 --2- 172.21.15.23:0/2710107099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10441016b0 0x7f1044075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 >> 172.21.15.23:0/2710107099 conn(0x7f10440fcc00 msgr2=0x7f1044105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 shutdown_connections 2024-03-28T15:52:47.471 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.470+0000 7f1049c81640 1 -- 172.21.15.23:0/2710107099 wait complete. 2024-03-28T15:52:47.482 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 -- 172.21.15.23:0/1684488908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 msgr2=0x7f0bd8106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1684488908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8106f80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009860 tx=0x7f0bc002bb70 comp rx=0 tx=0).stop 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 -- 172.21.15.23:0/1684488908 shutdown_connections 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1684488908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 0x7f0bd81098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1684488908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8106f80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1684488908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 -- 172.21.15.23:0/1684488908 >> 172.21.15.23:0/1684488908 conn(0x7f0bd80fbbd0 msgr2=0x7f0bd80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 -- 172.21.15.23:0/1684488908 shutdown_connections 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.585+0000 7f0bdd059640 1 -- 172.21.15.23:0/1684488908 wait complete. 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 Processor -- start 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 -- start start 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 0x7f0bd819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd810d320 con 0x7f0bd8102260 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd810d1a0 con 0x7f0bd8104b90 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bdd059640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd810d4a0 con 0x7f0bd81074c0 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44572/0 (socket says 172.21.15.23:44572) 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 -- 172.21.15.23:0/1089303495 learned_addr learned my addr 172.21.15.23:0/1089303495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bcffff640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd759e640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 0x7f0bd819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 -- 172.21.15.23:0/1089303495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 msgr2=0x7f0bd819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 0x7f0bd819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 -- 172.21.15.23:0/1089303495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 msgr2=0x7f0bd8197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bd6d9d640 1 -- 172.21.15.23:0/1089303495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd819f620 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.586+0000 7f0bcffff640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bd6d9d640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d950 tx=0x7f0bc800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8017070 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bc800de80 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc800f620 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd819f910 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.587+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd819fe80 con 0x7f0bd8102260 2024-03-28T15:52:47.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.588+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd80fd6c0 con 0x7f0bd8102260 2024-03-28T15:52:47.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.589+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0bc8009c30 con 0x7f0bd8102260 2024-03-28T15:52:47.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.590+0000 7f0bd4d99640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 0x7f0bc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.590+0000 7f0bcffff640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 0x7f0bc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.590+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f0bc80ce9c0 con 0x7f0bd8102260 2024-03-28T15:52:47.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.590+0000 7f0bcffff640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 0x7f0bc40a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009830 tx=0x7f0bc002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.598+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0bc80d0e20 con 0x7f0bd8102260 2024-03-28T15:52:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:47 smithi023 ceph-mon[32448]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 240 B/s, 1 objects/s recovering 2024-03-28T15:52:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2794361647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3345263443' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1840419634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.821 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.820+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd802d070 con 0x7f0bd8102260 2024-03-28T15:52:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:47 smithi117 ceph-mon[36002]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 240 B/s, 1 objects/s recovering 2024-03-28T15:52:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2794361647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3345263443' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1840419634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.821+0000 7f0bd4d99640 1 -- 172.21.15.23:0/1089303495 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f0bc809b670 con 0x7f0bd8102260 2024-03-28T15:52:47.822 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:47.822 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:52:47.823 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:47.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 msgr2=0x7f0bc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 0x7f0bc40a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0bc0009830 tx=0x7f0bc002e040 comp rx=0 tx=0).stop 2024-03-28T15:52:47.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 msgr2=0x7f0bd8197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f0bc800d950 tx=0x7f0bc800dd50 comp rx=0 tx=0).stop 2024-03-28T15:52:47.824 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 shutdown_connections 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f0bc40a2880 0x7f0bc40a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0bd81074c0 0x7f0bd819efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bd8104b90 0x7f0bd8197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 --2- 172.21.15.23:0/1089303495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0bd8102260 0x7f0bd8197a00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 >> 172.21.15.23:0/1089303495 conn(0x7f0bd80fbbd0 msgr2=0x7f0bd8102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.823+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 shutdown_connections 2024-03-28T15:52:47.825 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.824+0000 7f0bdd059640 1 -- 172.21.15.23:0/1089303495 wait complete. 2024-03-28T15:52:47.835 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:47 smithi120 ceph-mon[36826]: pgmap v360: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 240 B/s, 1 objects/s recovering 2024-03-28T15:52:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2794361647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3345263443' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1840419634' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:47.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 -- 172.21.15.23:0/315653354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 msgr2=0x7f4164106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 --2- 172.21.15.23:0/315653354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164106e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4154009900 tx=0x7f415402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 -- 172.21.15.23:0/315653354 shutdown_connections 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 --2- 172.21.15.23:0/315653354 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41641073a0 0x7f41641097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 --2- 172.21.15.23:0/315653354 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164106e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 --2- 172.21.15.23:0/315653354 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4164102140 0x7f4164104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 -- 172.21.15.23:0/315653354 >> 172.21.15.23:0/315653354 conn(0x7f41640fbb30 msgr2=0x7f41640fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 -- 172.21.15.23:0/315653354 shutdown_connections 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.938+0000 7f416c40b640 1 -- 172.21.15.23:0/315653354 wait complete. 2024-03-28T15:52:47.944 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 Processor -- start 2024-03-28T15:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 -- start start 2024-03-28T15:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 0x7f4164197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41641073a0 0x7f416419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d330 con 0x7f4164104a70 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d1b0 con 0x7f4164102140 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416c40b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416410d4b0 con 0x7f41641073a0 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44582/0 (socket says 172.21.15.23:44582) 2024-03-28T15:52:47.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 -- 172.21.15.23:0/1588927955 learned_addr learned my addr 172.21.15.23:0/1588927955 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416a1a7640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 0x7f4164197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416a9a8640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41641073a0 0x7f416419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 -- 172.21.15.23:0/1588927955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41641073a0 msgr2=0x7f416419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41641073a0 0x7f416419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 -- 172.21.15.23:0/1588927955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 msgr2=0x7f4164197a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:47.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 0x7f4164197a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f41699a6640 1 -- 172.21.15.23:0/1588927955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416419f620 con 0x7f4164104a70 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.939+0000 7f416a1a7640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 0x7f4164197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f41699a6640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f415402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154041070 con 0x7f4164104a70 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4154004450 con 0x7f4164104a70 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154034440 con 0x7f4164104a70 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416419f8b0 con 0x7f4164104a70 2024-03-28T15:52:47.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.940+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416419fda0 con 0x7f4164104a70 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.941+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41640fd620 con 0x7f4164104a70 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.942+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4154044070 con 0x7f4164104a70 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.943+0000 7f415b7fe640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 0x7f41500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.943+0000 7f416a1a7640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 0x7f41500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.943+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f41540f0b20 con 0x7f4164104a70 2024-03-28T15:52:47.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.943+0000 7f416a1a7640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 0x7f41500a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f41600095f0 tx=0x7f4160005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:47.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:47.947+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41540bd210 con 0x7f4164104a70 2024-03-28T15:52:48.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.130+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f416402d240 con 0x7f4164104a70 2024-03-28T15:52:48.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.132+0000 7f415b7fe640 1 -- 172.21.15.23:0/1588927955 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24913 (secure 0 0 0) 0x7f41540c12b0 con 0x7f4164104a70 2024-03-28T15:52:48.132 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:48.132 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":94,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:52:36.259332+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:48.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 msgr2=0x7f41500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 0x7f41500a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f41600095f0 tx=0x7f4160005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:48.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 msgr2=0x7f4164197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.134 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f415402c640 comp rx=0 tx=0).stop 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 shutdown_connections 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f41500a2800 0x7f41500a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f41641073a0 0x7f416419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4164104a70 0x7f4164197f40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 --2- 172.21.15.23:0/1588927955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4164102140 0x7f4164197a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 >> 172.21.15.23:0/1588927955 conn(0x7f41640fbb30 msgr2=0x7f4164102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 shutdown_connections 2024-03-28T15:52:48.135 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.134+0000 7f416c40b640 1 -- 172.21.15.23:0/1588927955 wait complete. 2024-03-28T15:52:48.148 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T15:52:48.149 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T15:52:48.284 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 -- 172.21.15.23:0/2948126991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c107420 msgr2=0x7fa47c109810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 --2- 172.21.15.23:0/2948126991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c107420 0x7fa47c109810 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa474009ca0 tx=0x7fa47402c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 -- 172.21.15.23:0/2948126991 shutdown_connections 2024-03-28T15:52:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 --2- 172.21.15.23:0/2948126991 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c107420 0x7fa47c109810 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.285 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 --2- 172.21.15.23:0/2948126991 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c104af0 0x7fa47c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 --2- 172.21.15.23:0/2948126991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 -- 172.21.15.23:0/2948126991 >> 172.21.15.23:0/2948126991 conn(0x7fa47c0fbb30 msgr2=0x7fa47c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 -- 172.21.15.23:0/2948126991 shutdown_connections 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 -- 172.21.15.23:0/2948126991 wait complete. 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.278+0000 7fa481990640 1 Processor -- start 2024-03-28T15:52:48.286 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 -- start start 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 0x7fa47c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c10d320 con 0x7fa47c104af0 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c10d1a0 con 0x7fa47c1021c0 2024-03-28T15:52:48.287 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa481990640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c10d4a0 con 0x7fa47c107420 2024-03-28T15:52:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:44600/0 (socket says 172.21.15.23:44600) 2024-03-28T15:52:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 -- 172.21.15.23:0/3901427329 learned_addr learned my addr 172.21.15.23:0/3901427329 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:48.288 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa47affd640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa47b7fe640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 0x7fa47c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 -- 172.21.15.23:0/3901427329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 msgr2=0x7fa47c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 0x7fa47c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 -- 172.21.15.23:0/3901427329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 msgr2=0x7fa47c199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa4727fc640 1 -- 172.21.15.23:0/3901427329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c1a1840 con 0x7fa47c104af0 2024-03-28T15:52:48.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa47b7fe640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 0x7fa47c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:48.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.279+0000 7fa47affd640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:48.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa4727fc640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa46400e9c0 tx=0x7fa46400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.290 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46400cd80 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa464005500 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464010690 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c1a1b30 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.280+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c1a2080 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.281+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c108f10 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.283+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa4640046c0 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.283+0000 7fa48098e640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 0x7fa4680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.283+0000 7fa47affd640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 0x7fa4680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.283+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fa4640cf240 con 0x7fa47c104af0 2024-03-28T15:52:48.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.284+0000 7fa47affd640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 0x7fa4680a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa47c19b100 tx=0x7fa46c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.292 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.288+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa46409bef0 con 0x7fa47c104af0 2024-03-28T15:52:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2710107099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1089303495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1588927955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.498+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa47c02d290 con 0x7fa47c104af0 2024-03-28T15:52:48.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.498+0000 7fa48098e640 1 -- 172.21.15.23:0/3901427329 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa46409b7f0 con 0x7fa47c104af0 2024-03-28T15:52:48.498 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T15:52:48.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 msgr2=0x7fa4680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 0x7fa4680a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa47c19b100 tx=0x7fa46c005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 msgr2=0x7fa47c19a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fa46400e9c0 tx=0x7fa46400ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 shutdown_connections 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa4680a2800 0x7fa4680a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa47c107420 0x7fa47c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa47c104af0 0x7fa47c19a170 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 --2- 172.21.15.23:0/3901427329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa47c1021c0 0x7fa47c199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 >> 172.21.15.23:0/3901427329 conn(0x7fa47c0fbb30 msgr2=0x7fa47c1019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 shutdown_connections 2024-03-28T15:52:48.501 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.500+0000 7fa481990640 1 -- 172.21.15.23:0/3901427329 wait complete. 2024-03-28T15:52:48.512 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.617+0000 7f26c5830640 1 -- 172.21.15.23:0/2673995720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 msgr2=0x7f26c01089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.617+0000 7f26c5830640 1 --2- 172.21.15.23:0/2673995720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c01089e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f26b8009900 tx=0x7f26b802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- 172.21.15.23:0/2673995720 shutdown_connections 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- 172.21.15.23:0/2673995720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c01089e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- 172.21.15.23:0/2673995720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.621 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- 172.21.15.23:0/2673995720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 0x7f26c01009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- 172.21.15.23:0/2673995720 >> 172.21.15.23:0/2673995720 conn(0x7f26c00fbb30 msgr2=0x7f26c00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- 172.21.15.23:0/2673995720 shutdown_connections 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- 172.21.15.23:0/2673995720 wait complete. 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 Processor -- start 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- start start 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 0x7f26c0197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c010c230 con 0x7f26c0102150 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c010c0b0 con 0x7f26c0101790 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.618+0000 7f26c5830640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26c010c3b0 con 0x7f26c01005a0 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.622 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55868/0 (socket says 172.21.15.23:55868) 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26be7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 -- 172.21.15.23:0/2302292694 learned_addr learned my addr 172.21.15.23:0/2302292694 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26beffd640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 0x7f26c0197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 -- 172.21.15.23:0/2302292694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 msgr2=0x7f26c0197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 0x7f26c0197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 -- 172.21.15.23:0/2302292694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 msgr2=0x7f26c0197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 -- 172.21.15.23:0/2302292694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c019f5b0 con 0x7f26c0102150 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26be7fc640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26bf7fe640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f26b8005ab0 tx=0x7f26b802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8041070 con 0x7f26c0102150 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26b8004450 con 0x7f26c0102150 2024-03-28T15:52:48.623 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26c019f840 con 0x7f26c0102150 2024-03-28T15:52:48.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26b8003e60 con 0x7f26c0102150 2024-03-28T15:52:48.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.619+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26c019fd30 con 0x7f26c0102150 2024-03-28T15:52:48.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.620+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26c019ff90 con 0x7f26c0102150 2024-03-28T15:52:48.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.621+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f26b8044070 con 0x7f26c0102150 2024-03-28T15:52:48.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.622+0000 7f26c482e640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 0x7f26ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.622+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f26b80f0e30 con 0x7f26c0102150 2024-03-28T15:52:48.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.622+0000 7f26beffd640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 0x7f26ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.622+0000 7f26beffd640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 0x7f26ac0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f26c01015f0 tx=0x7f26b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.628+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f26b80bd490 con 0x7f26c0102150 2024-03-28T15:52:48.740 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2710107099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:48.740 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1089303495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.740 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1588927955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.810+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26c01a0280 con 0x7f26c0102150 2024-03-28T15:52:48.811 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.811+0000 7f26c482e640 1 -- 172.21.15.23:0/2302292694 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f26b80c1530 con 0x7f26c0102150 2024-03-28T15:52:48.811 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 msgr2=0x7f26ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 0x7f26ac0a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f26c01015f0 tx=0x7f26b0009500 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 msgr2=0x7f26c019ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f26b8005ab0 tx=0x7f26b802c6c0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 shutdown_connections 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f26ac0a2800 0x7f26ac0a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f26c0102150 0x7f26c019ef40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f26c0101790 0x7f26c0197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 --2- 172.21.15.23:0/2302292694 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f26c01005a0 0x7f26c0197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.813 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.812+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 >> 172.21.15.23:0/2302292694 conn(0x7f26c00fbb30 msgr2=0x7f26c0068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.813+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 shutdown_connections 2024-03-28T15:52:48.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.813+0000 7f26c5830640 1 -- 172.21.15.23:0/2302292694 wait complete. 2024-03-28T15:52:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2710107099' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1089303495' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1588927955' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:48.824 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 -- 172.21.15.23:0/763332536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 msgr2=0x7fa324104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 --2- 172.21.15.23:0/763332536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324104520 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa314009900 tx=0x7fa31402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 -- 172.21.15.23:0/763332536 shutdown_connections 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 --2- 172.21.15.23:0/763332536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa3241097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 --2- 172.21.15.23:0/763332536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 0x7fa324106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 --2- 172.21.15.23:0/763332536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324104520 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 -- 172.21.15.23:0/763332536 >> 172.21.15.23:0/763332536 conn(0x7fa3240fbb40 msgr2=0x7fa3240fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 -- 172.21.15.23:0/763332536 shutdown_connections 2024-03-28T15:52:48.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.929+0000 7fa329590640 1 -- 172.21.15.23:0/763332536 wait complete. 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 Processor -- start 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 -- start start 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 0x7fa324197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa32419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32410d340 con 0x7fa324102130 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32410d1c0 con 0x7fa324107390 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa329590640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32410d4c0 con 0x7fa324104a60 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa322ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa322ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55894/0 (socket says 172.21.15.23:55894) 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa322ffd640 1 -- 172.21.15.23:0/585403771 learned_addr learned my addr 172.21.15.23:0/585403771 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa3227fc640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 0x7fa324197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.930+0000 7fa3237fe640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa32419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 -- 172.21.15.23:0/585403771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 msgr2=0x7fa324197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 0x7fa324197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 -- 172.21.15.23:0/585403771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 msgr2=0x7fa32419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa32419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 -- 172.21.15.23:0/585403771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32419f6a0 con 0x7fa324102130 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa3237fe640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa32419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa322ffd640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa314009230 tx=0x7fa31402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314041070 con 0x7fa324102130 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32419f930 con 0x7fa324102130 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa314004450 con 0x7fa324102130 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa314003db0 con 0x7fa324102130 2024-03-28T15:52:48.934 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.931+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32419fe20 con 0x7fa324102130 2024-03-28T15:52:48.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.932+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3240fd630 con 0x7fa324102130 2024-03-28T15:52:48.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.933+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa314044070 con 0x7fa324102130 2024-03-28T15:52:48.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.934+0000 7fa31bfff640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 0x7fa3100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:48.935 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.934+0000 7fa3227fc640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 0x7fa3100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:48.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.934+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fa3140f09d0 con 0x7fa324102130 2024-03-28T15:52:48.936 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.934+0000 7fa3227fc640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 0x7fa3100a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa324198f90 tx=0x7fa30c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:48.938 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:48.939+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3140bd680 con 0x7fa324102130 2024-03-28T15:52:49.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.139+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3240622a0 con 0x7fa324102130 2024-03-28T15:52:49.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.139+0000 7fa31bfff640 1 -- 172.21.15.23:0/585403771 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v94) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3140bcf80 con 0x7fa324102130 2024-03-28T15:52:49.139 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T15:52:49.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 msgr2=0x7fa3100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.141 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 0x7fa3100a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fa324198f90 tx=0x7fa30c005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 msgr2=0x7fa324197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa314009230 tx=0x7fa31402c650 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 shutdown_connections 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fa3100a2800 0x7fa3100a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa324107390 0x7fa32419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa324104a60 0x7fa324197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 --2- 172.21.15.23:0/585403771 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa324102130 0x7fa324197a80 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 >> 172.21.15.23:0/585403771 conn(0x7fa3240fbb40 msgr2=0x7fa324102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 shutdown_connections 2024-03-28T15:52:49.142 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.141+0000 7fa329590640 1 -- 172.21.15.23:0/585403771 wait complete. 2024-03-28T15:52:49.156 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 -- 172.21.15.23:0/399423141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4102150 msgr2=0x7fcbe41089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 --2- 172.21.15.23:0/399423141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4102150 0x7fcbe41089e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009ca0 tx=0x7fcbdc02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 -- 172.21.15.23:0/399423141 shutdown_connections 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 --2- 172.21.15.23:0/399423141 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4102150 0x7fcbe41089e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 --2- 172.21.15.23:0/399423141 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4101790 0x7fcbe4101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 --2- 172.21.15.23:0/399423141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe41009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 -- 172.21.15.23:0/399423141 >> 172.21.15.23:0/399423141 conn(0x7fcbe40fbb30 msgr2=0x7fcbe40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:49.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 -- 172.21.15.23:0/399423141 shutdown_connections 2024-03-28T15:52:49.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.264+0000 7fcbe999d640 1 -- 172.21.15.23:0/399423141 wait complete. 2024-03-28T15:52:49.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 Processor -- start 2024-03-28T15:52:49.266 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 -- start start 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4101790 0x7fcbe419c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410c230 con 0x7fcbe4102150 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410c0b0 con 0x7fcbe41005a0 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.265+0000 7fcbe999d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe410c3b0 con 0x7fcbe4101790 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe2ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47380/0 (socket says 172.21.15.23:47380) 2024-03-28T15:52:49.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55922/0 (socket says 172.21.15.23:55922) 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe2ffd640 1 -- 172.21.15.23:0/2892680431 learned_addr learned my addr 172.21.15.23:0/2892680431 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbdbfff640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4101790 0x7fcbe419c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 -- 172.21.15.23:0/2892680431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4101790 msgr2=0x7fcbe419c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4101790 0x7fcbe419c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 -- 172.21.15.23:0/2892680431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 msgr2=0x7fcbe4069370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 -- 172.21.15.23:0/2892680431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe419f5d0 con 0x7fcbe4102150 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe2ffd640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe37fe640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc02c630 tx=0x7fcbdc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc041020 con 0x7fcbe4102150 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbdc034cd0 con 0x7fcbe4102150 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe419f800 con 0x7fcbe4102150 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc039040 con 0x7fcbe4102150 2024-03-28T15:52:49.269 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.266+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe419fcc0 con 0x7fcbe4102150 2024-03-28T15:52:49.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.267+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe41078c0 con 0x7fcbe4102150 2024-03-28T15:52:49.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.268+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcbdc044020 con 0x7fcbe4102150 2024-03-28T15:52:49.270 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.269+0000 7fcbe899b640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 0x7fcbd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.269+0000 7fcbe2ffd640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 0x7fcbd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.269+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fcbdc0f0bf0 con 0x7fcbe4102150 2024-03-28T15:52:49.271 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.270+0000 7fcbe2ffd640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 0x7fcbd00a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40058c0 tx=0x7fcbd4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:49.274 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.274+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcbdc04b020 con 0x7fcbe4102150 2024-03-28T15:52:49.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:49 smithi023 ceph-mon[32448]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1 op/s; 202 B/s, 1 objects/s recovering 2024-03-28T15:52:49.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:49 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3901427329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:49 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2302292694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:49 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/585403771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.457 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.457+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcbe40622a0 con 0x7fcbe4102150 2024-03-28T15:52:49.458 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.458+0000 7fcbe899b640 1 -- 172.21.15.23:0/2892680431 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v94) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcbdc0bd930 con 0x7fcbe4102150 2024-03-28T15:52:49.458 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.459+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 msgr2=0x7fcbd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.459+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 0x7fcbd00a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcbd40058c0 tx=0x7fcbd4005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.459+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 msgr2=0x7fcbe419ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.459+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc02c630 tx=0x7fcbdc032f20 comp rx=0 tx=0).stop 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 shutdown_connections 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fcbd00a2800 0x7fcbd00a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fcbe4102150 0x7fcbe419ecc0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fcbe4101790 0x7fcbe419c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 --2- 172.21.15.23:0/2892680431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe41005a0 0x7fcbe4069370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 >> 172.21.15.23:0/2892680431 conn(0x7fcbe40fbb30 msgr2=0x7fcbe4104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:49.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 shutdown_connections 2024-03-28T15:52:49.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.460+0000 7fcbe999d640 1 -- 172.21.15.23:0/2892680431 wait complete. 2024-03-28T15:52:49.471 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 -- 172.21.15.23:0/475426928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 msgr2=0x7f7438101c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 --2- 172.21.15.23:0/475426928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438101c30 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7420009900 tx=0x7f742002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 -- 172.21.15.23:0/475426928 shutdown_connections 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 --2- 172.21.15.23:0/475426928 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 0x7f7438108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 --2- 172.21.15.23:0/475426928 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438101c30 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 --2- 172.21.15.23:0/475426928 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 0x7f74381009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 -- 172.21.15.23:0/475426928 >> 172.21.15.23:0/475426928 conn(0x7f74380fbb30 msgr2=0x7f74380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 -- 172.21.15.23:0/475426928 shutdown_connections 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.576+0000 7f743dfc6640 1 -- 172.21.15.23:0/475426928 wait complete. 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 Processor -- start 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 -- start start 2024-03-28T15:52:49.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 0x7f7438197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 0x7f743819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743810c240 con 0x7f74381017b0 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743810c0c0 con 0x7f7438102170 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f743dfc6640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f743810c3c0 con 0x7f74381005c0 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f7436ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f7436ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55946/0 (socket says 172.21.15.23:55946) 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f7436ffd640 1 -- 172.21.15.23:0/1144953151 learned_addr learned my addr 172.21.15.23:0/1144953151 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f74377fe640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 0x7f7438197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.577+0000 7f7437fff640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 0x7f743819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 -- 172.21.15.23:0/1144953151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 msgr2=0x7f7438197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 0x7f7438197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 -- 172.21.15.23:0/1144953151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 msgr2=0x7f743819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 0x7f743819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 -- 172.21.15.23:0/1144953151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743819f640 con 0x7f74381017b0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7436ffd640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7420009230 tx=0x7f742002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420041070 con 0x7f74381017b0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743819f8d0 con 0x7f74381017b0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7420004450 con 0x7f74381017b0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7420003e20 con 0x7f74381017b0 2024-03-28T15:52:49.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.578+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f743819fe10 con 0x7f74381017b0 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.579+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74381a0040 con 0x7f74381017b0 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.580+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7420044070 con 0x7f74381017b0 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.581+0000 7f7434ff9640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 0x7f74240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.581+0000 7f74377fe640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 0x7f74240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.581+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f74200f0d20 con 0x7f74381017b0 2024-03-28T15:52:49.582 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.581+0000 7f74377fe640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 0x7f74240a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7438101610 tx=0x7f7428009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:49.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.587+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f74200bd380 con 0x7f74381017b0 2024-03-28T15:52:49.722 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:49 smithi117 ceph-mon[36002]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1 op/s; 202 B/s, 1 objects/s recovering 2024-03-28T15:52:49.722 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:49 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3901427329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.722 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:49 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2302292694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.722 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:49 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/585403771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:49 smithi120 ceph-mon[36826]: pgmap v361: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1 op/s; 202 B/s, 1 objects/s recovering 2024-03-28T15:52:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:49 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3901427329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:49 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2302292694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:49 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/585403771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T15:52:49.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.929+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74381a0210 con 0x7f74381017b0 2024-03-28T15:52:49.929 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.929+0000 7f7434ff9640 1 -- 172.21.15.23:0/1144953151 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f74200c1420 con 0x7f74381017b0 2024-03-28T15:52:49.930 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:49.930 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:51:00.222875+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[24481,24494,24455,24454,14640]},"id":1}]} 2024-03-28T15:52:49.930 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 28 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 msgr2=0x7f74240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 0x7f74240a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7438101610 tx=0x7f7428009500 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 msgr2=0x7f7438197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7420009230 tx=0x7f742002c650 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 shutdown_connections 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f74240a2800 0x7f74240a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7438102170 0x7f743819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f74381017b0 0x7f7438197fd0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 --2- 172.21.15.23:0/1144953151 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f74381005c0 0x7f7438197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.931+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 >> 172.21.15.23:0/1144953151 conn(0x7f74380fbb30 msgr2=0x7f74380fd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.932+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 shutdown_connections 2024-03-28T15:52:49.932 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:49.932+0000 7f743dfc6640 1 -- 172.21.15.23:0/1144953151 wait complete. 2024-03-28T15:52:49.943 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T15:52:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 -- 172.21.15.23:0/652022459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 msgr2=0x7f60941009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 --2- 172.21.15.23:0/652022459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f60941009e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6084009900 tx=0x7f608402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 -- 172.21.15.23:0/652022459 shutdown_connections 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 --2- 172.21.15.23:0/652022459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 0x7f6094108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 --2- 172.21.15.23:0/652022459 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 0x7f6094101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 --2- 172.21.15.23:0/652022459 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f60941009e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 -- 172.21.15.23:0/652022459 >> 172.21.15.23:0/652022459 conn(0x7f60940fbb30 msgr2=0x7f60940fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 -- 172.21.15.23:0/652022459 shutdown_connections 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 -- 172.21.15.23:0/652022459 wait complete. 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.067+0000 7f609a626640 1 Processor -- start 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 -- start start 2024-03-28T15:52:50.070 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 0x7f609419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 0x7f609419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c210 con 0x7f60941005c0 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c090 con 0x7f6094102170 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f609a626640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609410c390 con 0x7f60941017b0 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55956/0 (socket says 172.21.15.23:55956) 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 -- 172.21.15.23:0/1454836239 learned_addr learned my addr 172.21.15.23:0/1454836239 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f60937fe640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 0x7f609419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6098bc3640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 0x7f609419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 -- 172.21.15.23:0/1454836239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 msgr2=0x7f609419c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 0x7f609419c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 -- 172.21.15.23:0/1454836239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 msgr2=0x7f609419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.071 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 0x7f609419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 -- 172.21.15.23:0/1454836239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609419f660 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.068+0000 7f6093fff640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6084000c00 tx=0x7f608402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.069+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084041070 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.069+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6084004450 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.069+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609419f890 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.069+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6084034440 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.069+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609419fd80 con 0x7f60941005c0 2024-03-28T15:52:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.070+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60941079b0 con 0x7f60941005c0 2024-03-28T15:52:50.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.071+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6084044070 con 0x7f60941005c0 2024-03-28T15:52:50.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.071+0000 7f60917fa640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.071+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f60840f0b20 con 0x7f60941005c0 2024-03-28T15:52:50.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.072+0000 7f60937fe640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.072+0000 7f60937fe640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 0x7f607c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f60940690e0 tx=0x7f6088005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.075+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f60840bd210 con 0x7f60941005c0 2024-03-28T15:52:50.289 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.289+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f609402d240 con 0x7f60941005c0 2024-03-28T15:52:50.291 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.291+0000 7f60917fa640 1 -- 172.21.15.23:0/1454836239 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+24913 (secure 0 0 0) 0x7f60840c12b0 con 0x7f60941005c0 2024-03-28T15:52:50.291 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T15:52:50.291 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":94,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T15:52:36.259332+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"90","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 msgr2=0x7f607c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 0x7f607c0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f60940690e0 tx=0x7f6088005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 msgr2=0x7f6094069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f6084000c00 tx=0x7f608402c640 comp rx=0 tx=0).stop 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 shutdown_connections 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f607c0a2800 0x7f607c0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6094102170 0x7f609419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.293 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f60941017b0 0x7f609419c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 --2- 172.21.15.23:0/1454836239 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f60941005c0 0x7f6094069370 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 >> 172.21.15.23:0/1454836239 conn(0x7f60940fbb30 msgr2=0x7f6094104b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 shutdown_connections 2024-03-28T15:52:50.294 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.293+0000 7f609a626640 1 -- 172.21.15.23:0/1454836239 wait complete. 2024-03-28T15:52:50.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 28, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-28T15:50:31.785133+0000', 'modified': '2024-03-28T15:51:00.222875+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24494, 'mds_1': 24481, 'mds_2': 24455, 'mds_3': 24454, 'mds_4': 14640}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14628': {'gid': 14628, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.23:6839/1641251810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6838', 'nonce': 1641251810}, {'type': 'v1', 'addr': '172.21.15.23:6839', 'nonce': 1641251810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14640': {'gid': 14640, 'name': 'a', 'rank': 4, 'incarnation': 26, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.23:6841/910694674', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.23:6840', 'nonce': 910694674}, {'type': 'v1', 'addr': '172.21.15.23:6841', 'nonce': 910694674}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24392': {'gid': 24392, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6833/2721990059', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6832', 'nonce': 2721990059}, {'type': 'v1', 'addr': '172.21.15.117:6833', 'nonce': 2721990059}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24403': {'gid': 24403, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6833/515874083', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6832', 'nonce': 515874083}, {'type': 'v1', 'addr': '172.21.15.120:6833', 'nonce': 515874083}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24422': {'gid': 24422, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6835/3656482331', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 3656482331}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 3656482331}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24442': {'gid': 24442, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.120:6835/1615812668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6834', 'nonce': 1615812668}, {'type': 'v1', 'addr': '172.21.15.120:6835', 'nonce': 1615812668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24454': {'gid': 24454, 'name': 'c', 'rank': 3, 'incarnation': 24, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.120:6837/2763171574', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6836', 'nonce': 2763171574}, {'type': 'v1', 'addr': '172.21.15.120:6837', 'nonce': 2763171574}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24455': {'gid': 24455, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.117:6837/30575642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 30575642}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 30575642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.120:6839/4092312487', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.120:6838', 'nonce': 4092312487}, {'type': 'v1', 'addr': '172.21.15.120:6839', 'nonce': 4092312487}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24494': {'gid': 24494, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6839/3971000510', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3971000510}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3971000510}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24494, 'qdb_cluster': [24481, 24494, 24455, 24454, 14640]} 2024-03-28T15:52:50.305 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-28T15:52:50.305 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- 172.21.15.23:0/541260311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901016c0 msgr2=0x7f5090101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 --2- 172.21.15.23:0/541260311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901016c0 0x7f5090101ae0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5080009900 tx=0x7f508002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- 172.21.15.23:0/541260311 shutdown_connections 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 --2- 172.21.15.23:0/541260311 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5090103270 0x7f5090109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 --2- 172.21.15.23:0/541260311 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f50901028b0 0x7f5090102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 --2- 172.21.15.23:0/541260311 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901016c0 0x7f5090101ae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- 172.21.15.23:0/541260311 >> 172.21.15.23:0/541260311 conn(0x7f50900fcc10 msgr2=0x7f50900ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- 172.21.15.23:0/541260311 shutdown_connections 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- 172.21.15.23:0/541260311 wait complete. 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 Processor -- start 2024-03-28T15:52:50.412 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.410+0000 7f509547e640 1 -- start start 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 0x7f5090197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5090103270 0x7f509019efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509010d310 con 0x7f50901016c0 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509010d190 con 0x7f50901028b0 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509010d490 con 0x7f5090103270 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55966/0 (socket says 172.21.15.23:55966) 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 -- 172.21.15.23:0/227771515 learned_addr learned my addr 172.21.15.23:0/227771515 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508e7fc640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 0x7f5090197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508f7fe640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5090103270 0x7f509019efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 -- 172.21.15.23:0/227771515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5090103270 msgr2=0x7f509019efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5090103270 0x7f509019efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.413 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 -- 172.21.15.23:0/227771515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 msgr2=0x7f5090197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 0x7f5090197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 -- 172.21.15.23:0/227771515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f509019f610 con 0x7f50901016c0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508e7fc640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 0x7f5090197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f508effd640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f508002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f5077fff640 1 -- 172.21.15.23:0/227771515 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080041070 con 0x7f50901016c0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f5077fff640 1 -- 172.21.15.23:0/227771515 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5080004450 con 0x7f50901016c0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509019f8a0 con 0x7f50901016c0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f5077fff640 1 -- 172.21.15.23:0/227771515 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080034440 con 0x7f50901016c0 2024-03-28T15:52:50.414 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.411+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509019fd90 con 0x7f50901016c0 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5077fff640 1 -- 172.21.15.23:0/227771515 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5080044070 con 0x7f50901016c0 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 Processor -- start 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 -- start start 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dc740 0x5556300dcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dd130 0x5556300e1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x555630041c20 con 0x5556300e22a0 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562ffeaa40 con 0x5556300dd130 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5098066740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55563003dcc0 con 0x5556300dc740 2024-03-28T15:52:50.421 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f5077fff640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 0x7f50780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55970/0 (socket says 172.21.15.23:55970) 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508e7fc640 1 -- 172.21.15.23:0/2797209919 learned_addr learned my addr 172.21.15.23:0/2797209919 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508effd640 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dd130 0x5556300e1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508f7fe640 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dc740 0x5556300dcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.414+0000 7f508f7fe640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 0x7f50780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f5077fff640 1 -- 172.21.15.23:0/227771515 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f50800f0900 con 0x7f50901016c0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 -- 172.21.15.23:0/2797209919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dc740 msgr2=0x5556300dcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dc740 0x5556300dcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 -- 172.21.15.23:0/2797209919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dd130 msgr2=0x5556300e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dd130 0x5556300e1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 -- 172.21.15.23:0/2797209919 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55563000cf90 con 0x5556300e22a0 2024-03-28T15:52:50.422 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508e7fc640 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f50840114f0 tx=0x7f50840118f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b649f44b8966087f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f508f7fe640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 0x7f50780a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5556300dd990 tx=0x7f5088009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f50757fa640 1 -- 172.21.15.23:0/2797209919 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508401e070 con 0x5556300e22a0 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f50757fa640 1 -- 172.21.15.23:0/2797209919 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50840035f0 con 0x5556300e22a0 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.415+0000 7f50757fa640 1 -- 172.21.15.23:0/2797209919 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400f740 con 0x5556300e22a0 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- 172.21.15.23:0/2797209919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 msgr2=0x5556300e4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f50840114f0 tx=0x7f50840118f0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- 172.21.15.23:0/2797209919 shutdown_connections 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300e4730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dd130 0x5556300e1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 --2- 172.21.15.23:0/2797209919 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dc740 0x5556300dcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- 172.21.15.23:0/2797209919 >> 172.21.15.23:0/2797209919 conn(0x5556300a4a60 msgr2=0x5556300d80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- 172.21.15.23:0/2797209919 shutdown_connections 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- 172.21.15.23:0/2797209919 wait complete. 2024-03-28T15:52:50.423 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 -- start start 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.416+0000 7f5098066740 1 Processor -- start 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e6c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300e22a0 0x5556300ee200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300e5250 con 0x5556300dd130 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55562fffde90 con 0x5556300dc740 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300e53d0 con 0x5556300e22a0 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 10 client.0 _ll_register_callbacks cb 0x55562fd55ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55980/0 (socket says 172.21.15.23:55980) 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 -- 172.21.15.23:0/3825697575 learned_addr learned my addr 172.21.15.23:0/3825697575 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508effd640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.424 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508f7fe640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300e22a0 0x5556300ee200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 -- 172.21.15.23:0/3825697575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300e22a0 msgr2=0x5556300ee200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300e22a0 0x5556300ee200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 -- 172.21.15.23:0/3825697575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 msgr2=0x5556300e6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e6c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 -- 172.21.15.23:0/3825697575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55563000a9d0 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f508e7fc640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f508401c040 tx=0x7f5084004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f79981909fb03561 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5072ff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084027020 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f5084005250 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.417+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f50840035f0 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5084032050 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 10 client.15429 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55562fd562d0 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50840035f0 con 0x5556300dd130 2024-03-28T15:52:50.425 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5084004ce0 con 0x5556300dd130 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 10 client.15429 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 10 client.15429 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 20 client.15429 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x5556300f3180 0x5556300f2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 4 client.15429 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 --> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55563004f160 con 0x5556300f3180 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f508effd640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x5556300f3180 0x5556300f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f508effd640 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x5556300f3180 0x5556300f2410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cbc30cd9b5f864de server_cookie=c64e823b7f59739a in_seq=0 out_seq=0 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.418+0000 7f5072ff5640 10 client.15429 ms_handle_connect on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5072ff5640 1 -- 172.21.15.23:0/3825697575 <== mds.4 v2:172.21.15.23:6840/910694674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55563004f160 con 0x5556300f3180 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5072ff5640 10 client.15429 handle_command_reply: tid=0 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 client.15429 shutdown 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 msgr2=0x5556300e71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.426 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f508401c040 tx=0x7f5084004230 comp rx=0 tx=0).stop 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 shutdown_connections 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x5556300f3180 0x5556300f2410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300e22a0 0x5556300ee200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dd130 0x5556300e71c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5072ff5640 0 client.15429 ms_handle_reset on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 --2- 172.21.15.23:0/3825697575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 >> 172.21.15.23:0/3825697575 conn(0x5556300a4a60 msgr2=0x5556300d72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 shutdown_connections 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 1 -- 172.21.15.23:0/3825697575 wait complete. 2024-03-28T15:52:50.427 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.419+0000 7f5098066740 20 client.15429 trim_cache size 0 max 16384 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 Processor -- start 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 -- start start 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300e22a0 0x5556300d7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f29a0 con 0x5556300dc740 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f3000 con 0x5556300e22a0 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f5098066740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f3660 con 0x5556300dd130 2024-03-28T15:52:50.442 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f508effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f508effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55982/0 (socket says 172.21.15.23:55982) 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f508effd640 1 -- 172.21.15.23:0/2610645411 learned_addr learned my addr 172.21.15.23:0/2610645411 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f508f7fe640 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300e22a0 0x5556300d7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.436+0000 7f508e7fc640 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 -- 172.21.15.23:0/2610645411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 msgr2=0x5556300ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300ee1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 -- 172.21.15.23:0/2610645411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300e22a0 msgr2=0x5556300d7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300e22a0 0x5556300d7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 -- 172.21.15.23:0/2610645411 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55562fe206b0 con 0x5556300dc740 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f508effd640 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f50800bccf0 tx=0x7f508002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992c1b0b58991c7c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f5074ff9640 1 -- 172.21.15.23:0/2610645411 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508009be60 con 0x5556300dc740 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f5074ff9640 1 -- 172.21.15.23:0/2610645411 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5080004c80 con 0x5556300dc740 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.437+0000 7f5074ff9640 1 -- 172.21.15.23:0/2610645411 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508004b040 con 0x5556300dc740 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- 172.21.15.23:0/2610645411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 msgr2=0x5556300e4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f50800bccf0 tx=0x7f508002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- 172.21.15.23:0/2610645411 shutdown_connections 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300e22a0 0x5556300d7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- 172.21.15.23:0/2610645411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300dc740 0x5556300e4690 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- 172.21.15.23:0/2610645411 >> 172.21.15.23:0/2610645411 conn(0x5556300a4a60 msgr2=0x5556300de630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- 172.21.15.23:0/2610645411 shutdown_connections 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- 172.21.15.23:0/2610645411 wait complete. 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- start start 2024-03-28T15:52:50.444 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 Processor -- start 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f4320 con 0x5556300e22a0 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f3cc0 con 0x5556300dc740 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556300f4980 con 0x5556300dd130 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f5098066740 10 client.0 _ll_register_callbacks cb 0x7f508401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:37424/0 (socket says 172.21.15.23:37424) 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508f7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:55984/0 (socket says 172.21.15.23:55984) 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.438+0000 7f508e7fc640 1 -- 172.21.15.23:0/1918172457 learned_addr learned my addr 172.21.15.23:0/1918172457 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.445 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 -- 172.21.15.23:0/1918172457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 msgr2=0x5556300e9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 -- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 msgr2=0x5556300ecdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 -- 172.21.15.23:0/1918172457 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556300ed430 con 0x5556300dd130 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508f7fe640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f508e7fc640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f5084009d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b002b63449a84 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5072ff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084027040 con 0x5556300dd130 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5080004c80 con 0x5556300dd130 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f508400fe00 con 0x5556300dd130 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f508403f050 con 0x5556300dd130 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5098066740 10 client.25228 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.446 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55563014e1e0 con 0x5556300dd130 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.439+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5080004c80 con 0x5556300dd130 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f5084036430 con 0x5556300dd130 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 10 client.25228 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 10 client.25228 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 20 client.25228 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x555630152750 0x55563014e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 4 client.25228 mds_command: new command op to 14640 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 --> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5556300494b0 con 0x555630152750 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f508effd640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x555630152750 0x55563014e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5072ff5640 10 client.25228 ms_handle_connect on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f508effd640 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x555630152750 0x55563014e410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2163c38233e2af20 server_cookie=cf97bceea7905cc in_seq=0 out_seq=0 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5072ff5640 1 -- 172.21.15.23:0/1918172457 <== mds.4 v2:172.21.15.23:6840/910694674 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5556300494b0 con 0x555630152750 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5072ff5640 10 client.25228 handle_command_reply: tid=0 2024-03-28T15:52:50.447 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 client.25228 shutdown 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 msgr2=0x5556300e9f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5084000c00 tx=0x7f5084009d80 comp rx=0 tx=0).stop 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 shutdown_connections 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:6840/910694674,v1:172.21.15.23:6841/910694674] conn(0x555630152750 0x55563014e410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5556300e22a0 0x5556300ecdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5556300dd130 0x5556300e9f70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 --2- 172.21.15.23:0/1918172457 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5556300dc740 0x5556300e9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5072ff5640 0 client.25228 ms_handle_reset on v2:172.21.15.23:6840/910694674 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.440+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 >> 172.21.15.23:0/1918172457 conn(0x5556300a4a60 msgr2=0x5556300d7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 shutdown_connections 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f5098066740 1 -- 172.21.15.23:0/1918172457 wait complete. 2024-03-28T15:52:50.448 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f5098066740 20 client.25228 trim_cache size 0 max 16384 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 4, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 14640, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 111.221963246, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 94, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 145.32732036900001 2024-03-28T15:52:50.449 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:50.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 msgr2=0x7f50780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 0x7f50780a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x5556300dd990 tx=0x7f5088009500 comp rx=0 tx=0).stop 2024-03-28T15:52:50.450 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 msgr2=0x7f50901979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f5080000c00 tx=0x7f508002c640 comp rx=0 tx=0).stop 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 shutdown_connections 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f50780a2800 0x7f50780a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5090103270 0x7f509019efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50901028b0 0x7f5090197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 --2- 172.21.15.23:0/227771515 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f50901016c0 0x7f50901979f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 >> 172.21.15.23:0/227771515 conn(0x7f50900fcc10 msgr2=0x7f5090105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 shutdown_connections 2024-03-28T15:52:50.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.441+0000 7f509547e640 1 -- 172.21.15.23:0/227771515 wait complete. 2024-03-28T15:52:50.452 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 -- 172.21.15.23:0/3440224720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1734107440 msgr2=0x7f1734109830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 --2- 172.21.15.23:0/3440224720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1734107440 0x7f1734109830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f172c009ca0 tx=0x7f172c02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 -- 172.21.15.23:0/3440224720 shutdown_connections 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 --2- 172.21.15.23:0/3440224720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1734107440 0x7f1734109830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 --2- 172.21.15.23:0/3440224720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734104b10 0x7f1734106f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 --2- 172.21.15.23:0/3440224720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17341021e0 0x7f17341045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 -- 172.21.15.23:0/3440224720 >> 172.21.15.23:0/3440224720 conn(0x7f17340fbb50 msgr2=0x7f17340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.558 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 -- 172.21.15.23:0/3440224720 shutdown_connections 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.555+0000 7f173a075640 1 -- 172.21.15.23:0/3440224720 wait complete. 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 Processor -- start 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 -- start start 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 0x7f1734197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 0x7f173419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410d340 con 0x7f17341021e0 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410d1c0 con 0x7f1734104b10 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f173a075640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f173410d4c0 con 0x7f1734107440 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56006/0 (socket says 172.21.15.23:56006) 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.556+0000 7f17337fe640 1 -- 172.21.15.23:0/343478891 learned_addr learned my addr 172.21.15.23:0/343478891 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.559 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1732ffd640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 0x7f1734197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1733fff640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 0x7f173419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 -- 172.21.15.23:0/343478891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 msgr2=0x7f173419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 0x7f173419ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 -- 172.21.15.23:0/343478891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 msgr2=0x7f1734197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 0x7f1734197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 -- 172.21.15.23:0/343478891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f173419f5d0 con 0x7f17341021e0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1732ffd640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 0x7f1734197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1733fff640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 0x7f173419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f17337fe640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f172400ea60 tx=0x7f172400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1730ff9640 1 -- 172.21.15.23:0/343478891 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172400cde0 con 0x7f17341021e0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1730ff9640 1 -- 172.21.15.23:0/343478891 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17240035f0 con 0x7f17341021e0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f1730ff9640 1 -- 172.21.15.23:0/343478891 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724010670 con 0x7f17341021e0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f173419f8c0 con 0x7f17341021e0 2024-03-28T15:52:50.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.557+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f173419fe10 con 0x7f17341021e0 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.559+0000 7f1730ff9640 1 -- 172.21.15.23:0/343478891 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f17240037a0 con 0x7f17341021e0 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 Processor -- start 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f1730ff9640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 0x7f171c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 -- start start 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f1733fff640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 0x7f171c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f1730ff9640 1 -- 172.21.15.23:0/343478891 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f17240cf140 con 0x7f17341021e0 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e77c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e7a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329dd7c20 con 0x55d329e72740 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329d80a40 con 0x55d329e782a0 2024-03-28T15:52:50.567 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f173cc5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329dd3cc0 con 0x55d329e73130 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56014/0 (socket says 172.21.15.23:56014) 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 -- 172.21.15.23:0/1474729979 learned_addr learned my addr 172.21.15.23:0/1474729979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f1732ffd640 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e77c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f1733fff640 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e7a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 -- 172.21.15.23:0/1474729979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 msgr2=0x55d329e77c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e77c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 -- 172.21.15.23:0/1474729979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 msgr2=0x55d329e7a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e7a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.560+0000 7f17337fe640 1 -- 172.21.15.23:0/1474729979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d329da2f90 con 0x55d329e72740 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f17337fe640 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f17240a3960 tx=0x7f1724097ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61ae6d3e7e903dc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f1733fff640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 0x7f171c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c630 tx=0x7f172c02ceb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f1719ffb640 1 -- 172.21.15.23:0/1474729979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17240b4070 con 0x55d329e72740 2024-03-28T15:52:50.568 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f1719ffb640 1 -- 172.21.15.23:0/1474729979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f172409f840 con 0x55d329e72740 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f1719ffb640 1 -- 172.21.15.23:0/1474729979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1724097d60 con 0x55d329e72740 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 -- 172.21.15.23:0/1474729979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 msgr2=0x55d329e72b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f17240a3960 tx=0x7f1724097ac0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 -- 172.21.15.23:0/1474729979 shutdown_connections 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e7a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e77c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1474729979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e72b60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.561+0000 7f173cc5c740 1 -- 172.21.15.23:0/1474729979 >> 172.21.15.23:0/1474729979 conn(0x55d329e3aa60 msgr2=0x55d329e6e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- 172.21.15.23:0/1474729979 shutdown_connections 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- 172.21.15.23:0/1474729979 wait complete. 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- start start 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 Processor -- start 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.569 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e7d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e841e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e7b1f0 con 0x55d329e72740 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329d93e90 con 0x55d329e782a0 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e7b370 con 0x55d329e73130 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f173cc5c740 10 client.0 _ll_register_callbacks cb 0x55d329aebac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56026/0 (socket says 172.21.15.23:56026) 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f17337fe640 1 -- 172.21.15.23:0/3433589602 learned_addr learned my addr 172.21.15.23:0/3433589602 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.562+0000 7f1732ffd640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e7d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1733fff640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e841e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 -- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 msgr2=0x55d329e7d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e7d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.570 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 -- 172.21.15.23:0/3433589602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 msgr2=0x55d329e841e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e841e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 -- 172.21.15.23:0/3433589602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d329da09d0 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1732ffd640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e7d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f17337fe640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f172400c760 tx=0x7f1724003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59132c188f51bfd server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17240b8070 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f172409ab40 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f172409f840 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17240b4050 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f173cc5c740 10 client.15447 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d329aec2d0 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.563+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f172409f840 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f1724097560 con 0x55d329e72740 2024-03-28T15:52:50.571 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 10 client.15447 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 10 client.15447 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 20 client.15447 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329e88340 0x55d329e8d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 4 client.15447 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 --> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d329de5160 con 0x55d329e88340 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f1732ffd640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329e88340 0x55d329e8d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f1732ffd640 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329e88340 0x55d329e8d380 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d04da4d3f1e189 server_cookie=7574dd9cc131271c in_seq=0 out_seq=0 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.564+0000 7f1712ffd640 10 client.15447 ms_handle_connect on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f1712ffd640 1 -- 172.21.15.23:0/3433589602 <== mds.3 v2:172.21.15.120:6836/2763171574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d329de5160 con 0x55d329e88340 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f1712ffd640 10 client.15447 handle_command_reply: tid=0 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 client.15447 shutdown 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 msgr2=0x55d329e7cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f172400c760 tx=0x7f1724003ea0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.572 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 shutdown_connections 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329e88340 0x55d329e8d380 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e841e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e7d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f1712ffd640 0 client.15447 ms_handle_reset on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.565+0000 7f173cc5c740 1 --2- 172.21.15.23:0/3433589602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e7cc60 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.566+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 >> 172.21.15.23:0/3433589602 conn(0x55d329e3aa60 msgr2=0x55d329e6d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.566+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 shutdown_connections 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.566+0000 7f173cc5c740 1 -- 172.21.15.23:0/3433589602 wait complete. 2024-03-28T15:52:50.573 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.566+0000 7f173cc5c740 20 client.15447 trim_cache size 0 max 16384 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.582+0000 7f173cc5c740 1 Processor -- start 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.582+0000 7f173cc5c740 1 -- start start 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e72740 0x55d329e7cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e841a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e88720 con 0x55d329e782a0 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e88d80 con 0x55d329e72740 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f173cc5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e893e0 con 0x55d329e73130 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56040/0 (socket says 172.21.15.23:56040) 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 -- 172.21.15.23:0/1005207466 learned_addr learned my addr 172.21.15.23:0/1005207466 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1732ffd640 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e841a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f17337fe640 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e72740 0x55d329e7cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 -- 172.21.15.23:0/1005207466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 msgr2=0x55d329e841a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e841a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 -- 172.21.15.23:0/1005207466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e72740 msgr2=0x55d329e7cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e72740 0x55d329e7cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 -- 172.21.15.23:0/1005207466 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d329bb66b0 con 0x55d329e782a0 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f1733fff640 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f172c005e30 tx=0x7f172c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3e2928990a5d13 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f17197fa640 1 -- 172.21.15.23:0/1005207466 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c039040 con 0x55d329e782a0 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f17197fa640 1 -- 172.21.15.23:0/1005207466 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f172c052070 con 0x55d329e782a0 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.583+0000 7f17197fa640 1 -- 172.21.15.23:0/1005207466 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c0042a0 con 0x55d329e782a0 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- 172.21.15.23:0/1005207466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 msgr2=0x55d329e6ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f172c005e30 tx=0x7f172c02e050 comp rx=0 tx=0).stop 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- 172.21.15.23:0/1005207466 shutdown_connections 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e782a0 0x55d329e6ca60 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e841a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1005207466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e72740 0x55d329e7cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- 172.21.15.23:0/1005207466 >> 172.21.15.23:0/1005207466 conn(0x55d329e3aa60 msgr2=0x55d329e7c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- 172.21.15.23:0/1005207466 shutdown_connections 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- 172.21.15.23:0/1005207466 wait complete. 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 -- start start 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.584+0000 7f173cc5c740 1 Processor -- start 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e80b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e810a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e8a0a0 con 0x55d329e72740 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e89a40 con 0x55d329e782a0 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d329e8a700 con 0x55d329e73130 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 10 client.0 _ll_register_callbacks cb 0x7f17240b2070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56044/0 (socket says 172.21.15.23:56044) 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 -- 172.21.15.23:0/1998331579 learned_addr learned my addr 172.21.15.23:0/1998331579 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f1733fff640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e810a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f1732ffd640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e80b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 -- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 msgr2=0x55d329e80b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e80b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 -- 172.21.15.23:0/1998331579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 msgr2=0x55d329e810a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e810a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 -- 172.21.15.23:0/1998331579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d329ee44a0 con 0x55d329e72740 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f17337fe640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17240980c0 tx=0x7f1724097a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffd18a0bac39e403 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f1712ffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17240b8070 con 0x55d329e72740 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f172409ab40 con 0x55d329e72740 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.585+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f172c052070 con 0x55d329e72740 2024-03-28T15:52:50.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17240afdb0 con 0x55d329e72740 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 10 client.15453 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55d329ee44a0 con 0x55d329e72740 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f172c052070 con 0x55d329e72740 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f172409a5b0 con 0x55d329e72740 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 10 client.15453 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 10 client.15453 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 20 client.15453 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329ee9920 0x55d329ee5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 4 client.15453 mds_command: new command op to 24454 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 --> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d329ddf4b0 con 0x55d329ee9920 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1732ffd640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329ee9920 0x55d329ee5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1712ffd640 10 client.15453 ms_handle_connect on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.586+0000 7f1732ffd640 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329ee9920 0x55d329ee5780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=16f21d1986e0cf0f server_cookie=6abbb8196979f999 in_seq=0 out_seq=0 2024-03-28T15:52:50.593 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f1712ffd640 1 -- 172.21.15.23:0/1998331579 <== mds.3 v2:172.21.15.120:6836/2763171574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55d329ddf4b0 con 0x55d329ee9920 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f1712ffd640 10 client.15453 handle_command_reply: tid=0 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 client.15453 shutdown 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 msgr2=0x55d329e840c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17240980c0 tx=0x7f1724097a90 comp rx=0 tx=0).stop 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 shutdown_connections 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:6836/2763171574,v1:172.21.15.120:6837/2763171574] conn(0x55d329ee9920 0x55d329ee5780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d329e782a0 0x55d329e810a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55d329e73130 0x55d329e80b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 --2- 172.21.15.23:0/1998331579 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55d329e72740 0x55d329e840c0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 >> 172.21.15.23:0/1998331579 conn(0x55d329e3aa60 msgr2=0x55d329e712f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f1712ffd640 0 client.15453 ms_handle_reset on v2:172.21.15.120:6836/2763171574 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 shutdown_connections 2024-03-28T15:52:50.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 1 -- 172.21.15.23:0/1998331579 wait complete. 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173cc5c740 20 client.15453 trim_cache size 0 max 16384 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 3, 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24454, 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 112.01988417699999, 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:50.595 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 94, 2024-03-28T15:52:50.596 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:50.596 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 156.81777647199999 2024-03-28T15:52:50.596 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:50.596 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 msgr2=0x7f171c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 0x7f171c0a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f172c02c630 tx=0x7f172c02ceb0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 msgr2=0x7f17341979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.587+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f172400ea60 tx=0x7f172400ef20 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 shutdown_connections 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f171c0a2800 0x7f171c0a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1734107440 0x7f173419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1734104b10 0x7f1734197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 --2- 172.21.15.23:0/343478891 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f17341021e0 0x7f17341979b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 >> 172.21.15.23:0/343478891 conn(0x7f17340fbb50 msgr2=0x7f1734102b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 shutdown_connections 2024-03-28T15:52:50.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.588+0000 7f173a075640 1 -- 172.21.15.23:0/343478891 wait complete. 2024-03-28T15:52:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2892680431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1144953151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1454836239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.599 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-28T15:52:50.703 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 -- 172.21.15.23:0/1496365940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150102150 msgr2=0x7fd1501089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 --2- 172.21.15.23:0/1496365940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150102150 0x7fd1501089e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd148009be0 tx=0x7fd14802bef0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 -- 172.21.15.23:0/1496365940 shutdown_connections 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 --2- 172.21.15.23:0/1496365940 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150102150 0x7fd1501089e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 --2- 172.21.15.23:0/1496365940 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150101790 0x7fd150101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 --2- 172.21.15.23:0/1496365940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 0x7fd1501009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 -- 172.21.15.23:0/1496365940 >> 172.21.15.23:0/1496365940 conn(0x7fd1500fbb30 msgr2=0x7fd1500fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 -- 172.21.15.23:0/1496365940 shutdown_connections 2024-03-28T15:52:50.704 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.703+0000 7fd156d37640 1 -- 172.21.15.23:0/1496365940 wait complete. 2024-03-28T15:52:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 Processor -- start 2024-03-28T15:52:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 -- start start 2024-03-28T15:52:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 0x7fd150196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.706 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150101790 0x7fd150196db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010c250 con 0x7fd150102150 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010c0d0 con 0x7fd1501005a0 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd156d37640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15010c3d0 con 0x7fd150101790 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56068/0 (socket says 172.21.15.23:56068) 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd154ad3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 0x7fd150196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 -- 172.21.15.23:0/3659662431 learned_addr learned my addr 172.21.15.23:0/3659662431 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150101790 0x7fd150196db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 -- 172.21.15.23:0/3659662431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150101790 msgr2=0x7fd150196db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150101790 0x7fd150196db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 -- 172.21.15.23:0/3659662431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 msgr2=0x7fd150196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.707 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 0x7fd150196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.704+0000 7fd1552d4640 1 -- 172.21.15.23:0/3659662431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15019e490 con 0x7fd150102150 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd1552d4640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd148036040 tx=0x7fd1480337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd14dffb640 1 -- 172.21.15.23:0/3659662431 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148043020 con 0x7fd150102150 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15019e720 con 0x7fd150102150 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd14dffb640 1 -- 172.21.15.23:0/3659662431 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd14802ca60 con 0x7fd150102150 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd14dffb640 1 -- 172.21.15.23:0/3659662431 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148034690 con 0x7fd150102150 2024-03-28T15:52:50.708 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.705+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15019ec70 con 0x7fd150102150 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 Processor -- start 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd14dffb640 1 -- 172.21.15.23:0/3659662431 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd148046020 con 0x7fd150102150 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 -- start start 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303a6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303abe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303ae940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb30309c20 con 0x55eb303a6950 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb302b2a40 con 0x55eb303ac4b0 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd1599f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb30305cc0 con 0x55eb303a7340 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303a6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303a6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56084/0 (socket says 172.21.15.23:56084) 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd14ffff640 1 -- 172.21.15.23:0/3695952916 learned_addr learned my addr 172.21.15.23:0/3695952916 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.708+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd14dffb640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 0x7fd1400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 -- 172.21.15.23:0/3695952916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 msgr2=0x55eb303ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 -- 172.21.15.23:0/3695952916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 msgr2=0x55eb303a6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303a6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 -- 172.21.15.23:0/3695952916 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb302d4f90 con 0x55eb303a7340 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd14dffb640 1 -- 172.21.15.23:0/3659662431 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fd1480f13b0 con 0x7fd150102150 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd1552d4640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 0x7fd1400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303abe40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd144009bf0 tx=0x7fd144011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd2dda5bc627064 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd13affd640 1 -- 172.21.15.23:0/3695952916 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144009da0 con 0x55eb303a7340 2024-03-28T15:52:50.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd1552d4640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 0x7fd1400a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd148006010 tx=0x7fd1480a6f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd13affd640 1 -- 172.21.15.23:0/3695952916 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd144005020 con 0x55eb303a7340 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.709+0000 7fd13affd640 1 -- 172.21.15.23:0/3695952916 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144004b60 con 0x55eb303a7340 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- 172.21.15.23:0/3695952916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 msgr2=0x55eb303abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303abe40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd144009bf0 tx=0x7fd144011810 comp rx=0 tx=0).stop 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- 172.21.15.23:0/3695952916 shutdown_connections 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303abe40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3695952916 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303a6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- 172.21.15.23:0/3695952916 >> 172.21.15.23:0/3695952916 conn(0x55eb3036ca60 msgr2=0x55eb303a00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- 172.21.15.23:0/3695952916 shutdown_connections 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- 172.21.15.23:0/3695952916 wait complete. 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 -- start start 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.710+0000 7fd1599f6740 1 Processor -- start 2024-03-28T15:52:50.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303af460 con 0x55eb303a6950 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb302c5e90 con 0x55eb303ac4b0 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303af5e0 con 0x55eb303a7340 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 10 client.0 _ll_register_callbacks cb 0x55eb3001dac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56088/0 (socket says 172.21.15.23:56088) 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd14ffff640 1 -- 172.21.15.23:0/3532689227 learned_addr learned my addr 172.21.15.23:0/3532689227 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 -- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 msgr2=0x55eb303b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 -- 172.21.15.23:0/3532689227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 msgr2=0x55eb303b0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 -- 172.21.15.23:0/3532689227 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb302d29d0 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd154ad3640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b1410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55eb303a1000 tx=0x7fd144009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3cf09d06be253fe server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd12ffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144029020 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fd144004ab0 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.711+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd144005020 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd144034050 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 10 client.25258 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eb3001e2d0 con 0x55eb303a7340 2024-03-28T15:52:50.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd144005020 con 0x55eb303a7340 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd144019450 con 0x55eb303a7340 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 10 client.25258 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 10 client.25258 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 20 client.25258 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb303bd3e0 0x55eb303bc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 4 client.25258 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 --> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb30317160 con 0x55eb303bd3e0 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.712+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb303bd3e0 0x55eb303bc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd12ffff640 10 client.25258 ms_handle_connect on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd14ffff640 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb303bd3e0 0x55eb303bc6c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b0eb4b948bb39917 server_cookie=c73a15534fb6ffd6 in_seq=0 out_seq=0 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd12ffff640 1 -- 172.21.15.23:0/3532689227 <== mds.2 v2:172.21.15.117:6836/30575642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55eb30317160 con 0x55eb303bd3e0 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd12ffff640 10 client.25258 handle_command_reply: tid=0 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd1599f6740 1 client.25258 shutdown 2024-03-28T15:52:50.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 msgr2=0x55eb303b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.713+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b1410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x55eb303a1000 tx=0x7fd144009fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 shutdown_connections 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb303bd3e0 0x55eb303bc6c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd12ffff640 0 client.25258 ms_handle_reset on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b1410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 --2- 172.21.15.23:0/3532689227 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a6950 0x55eb303b0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 >> 172.21.15.23:0/3532689227 conn(0x55eb3036ca60 msgr2=0x55eb3039f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 shutdown_connections 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 1 -- 172.21.15.23:0/3532689227 wait complete. 2024-03-28T15:52:50.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.714+0000 7fd1599f6740 20 client.25258 trim_cache size 0 max 16384 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 Processor -- start 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 -- start start 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303a6950 0x55eb303a32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303bcc00 con 0x55eb303ac4b0 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303bd260 con 0x55eb303a6950 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd1599f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303bd8c0 con 0x55eb303a7340 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.742 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd14ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56100/0 (socket says 172.21.15.23:56100) 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd14ffff640 1 -- 172.21.15.23:0/1154090576 learned_addr learned my addr 172.21.15.23:0/1154090576 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd154ad3640 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.736+0000 7fd14ffff640 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303a6950 0x55eb303a32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 -- 172.21.15.23:0/1154090576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 msgr2=0x55eb303b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 -- 172.21.15.23:0/1154090576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303a6950 msgr2=0x55eb303a32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303a6950 0x55eb303a32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 -- 172.21.15.23:0/1154090576 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb303bca30 con 0x55eb303ac4b0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd14ffff640 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5657f33df71284a server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd13a7fc640 1 -- 172.21.15.23:0/1154090576 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c0272f0 con 0x55eb303ac4b0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd13a7fc640 1 -- 172.21.15.23:0/1154090576 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd13c010800 con 0x55eb303ac4b0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.737+0000 7fd13a7fc640 1 -- 172.21.15.23:0/1154090576 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003970 con 0x55eb303ac4b0 2024-03-28T15:52:50.743 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- 172.21.15.23:0/1154090576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 msgr2=0x55eb303a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c0046c0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- 172.21.15.23:0/1154090576 shutdown_connections 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303ac4b0 0x55eb303a3830 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a7340 0x55eb303b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- 172.21.15.23:0/1154090576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303a6950 0x55eb303a32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- 172.21.15.23:0/1154090576 >> 172.21.15.23:0/1154090576 conn(0x55eb3036ca60 msgr2=0x55eb303a88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- 172.21.15.23:0/1154090576 shutdown_connections 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- 172.21.15.23:0/1154090576 wait complete. 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- start start 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 Processor -- start 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a6950 0x55eb303b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.744 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303be580 con 0x55eb303a7340 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303bdf20 con 0x55eb303ac4b0 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb303bebe0 con 0x55eb303a6950 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd1599f6740 10 client.0 _ll_register_callbacks cb 0x7fd13c009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.738+0000 7fd154ad3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56116/0 (socket says 172.21.15.23:56116) 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 -- 172.21.15.23:0/2023481439 learned_addr learned my addr 172.21.15.23:0/2023481439 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd14ffff640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd14ffff640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a6950 0x55eb303b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 -- 172.21.15.23:0/2023481439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a6950 msgr2=0x55eb303b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a6950 0x55eb303b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 -- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 msgr2=0x55eb303b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.745 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 -- 172.21.15.23:0/2023481439 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb300e86b0 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd14ffff640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd12ffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd154ad3640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd144005590 tx=0x7fd144009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31be2cb031d7b132 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd144029040 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd13c010800 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fd144019a60 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd144040050 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd1599f6740 10 client.15471 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.739+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55eb30418f20 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd13c010800 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fd144004dd0 con 0x55eb303a7340 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 10 client.15471 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.746 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 10 client.15471 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 20 client.15471 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb3041d4b0 0x55eb30419150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 4 client.15471 mds_command: new command op to 24455 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 --> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eb303114b0 con 0x55eb3041d4b0 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd14ffff640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb3041d4b0 0x55eb30419150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd14ffff640 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb3041d4b0 0x55eb30419150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d48775fbc08b7956 server_cookie=ff04a9a59a2cd3b5 in_seq=0 out_seq=0 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.740+0000 7fd12ffff640 10 client.15471 ms_handle_connect on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd12ffff640 1 -- 172.21.15.23:0/2023481439 <== mds.2 v2:172.21.15.117:6836/30575642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x55eb303114b0 con 0x55eb3041d4b0 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd12ffff640 10 client.15471 handle_command_reply: tid=0 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 client.15471 shutdown 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 msgr2=0x55eb303b8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd144005590 tx=0x7fd144009b60 comp rx=0 tx=0).stop 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 shutdown_connections 2024-03-28T15:52:50.747 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:6836/30575642,v1:172.21.15.117:6837/30575642] conn(0x55eb3041d4b0 0x55eb30419150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb303ac4b0 0x55eb303b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd12ffff640 0 client.15471 ms_handle_reset on v2:172.21.15.117:6836/30575642 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55eb303a7340 0x55eb303b8270 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 --2- 172.21.15.23:0/2023481439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55eb303a6950 0x55eb303b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 >> 172.21.15.23:0/2023481439 conn(0x55eb3036ca60 msgr2=0x55eb303a32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 shutdown_connections 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 1 -- 172.21.15.23:0/2023481439 wait complete. 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.741+0000 7fd1599f6740 20 client.15471 trim_cache size 0 max 16384 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:50.748 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 2, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24455, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 113.18306882, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 94, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 154.052268626 2024-03-28T15:52:50.749 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 msgr2=0x7fd1400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 0x7fd1400a4cc0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd148006010 tx=0x7fd1480a6f90 comp rx=0 tx=0).stop 2024-03-28T15:52:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 msgr2=0x7fd15019de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd148036040 tx=0x7fd1480337b0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 shutdown_connections 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fd1400a2800 0x7fd1400a4cc0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fd150102150 0x7fd15019de20 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fd150101790 0x7fd150196db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 --2- 172.21.15.23:0/3659662431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1501005a0 0x7fd150196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 >> 172.21.15.23:0/3659662431 conn(0x7fd1500fbb30 msgr2=0x7fd150104a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 shutdown_connections 2024-03-28T15:52:50.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.742+0000 7fd156d37640 1 -- 172.21.15.23:0/3659662431 wait complete. 2024-03-28T15:52:50.753 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-28T15:52:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2892680431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1144953151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1454836239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2892680431' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T15:52:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1144953151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1454836239' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T15:52:50.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 -- 172.21.15.23:0/237127944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec0fff70 msgr2=0x7efdec100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 --2- 172.21.15.23:0/237127944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec0fff70 0x7efdec100390 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efde0009900 tx=0x7efde002bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:50.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 -- 172.21.15.23:0/237127944 shutdown_connections 2024-03-28T15:52:50.861 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 --2- 172.21.15.23:0/237127944 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 0x7efdec1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 --2- 172.21.15.23:0/237127944 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec1008d0 0x7efdec100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 --2- 172.21.15.23:0/237127944 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec0fff70 0x7efdec100390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 -- 172.21.15.23:0/237127944 >> 172.21.15.23:0/237127944 conn(0x7efdec0fbb30 msgr2=0x7efdec0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 -- 172.21.15.23:0/237127944 shutdown_connections 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.859+0000 7efdf1e80640 1 -- 172.21.15.23:0/237127944 wait complete. 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 Processor -- start 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 -- start start 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 0x7efdec19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 0x7efdec1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec10f530 con 0x7efdec0fff70 2024-03-28T15:52:50.862 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec10f3b0 con 0x7efdec101270 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdf1e80640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec10f6b0 con 0x7efdec1008d0 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeaffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 0x7efdec19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56124/0 (socket says 172.21.15.23:56124) 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 -- 172.21.15.23:0/3805256156 learned_addr learned my addr 172.21.15.23:0/3805256156 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdebfff640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 0x7efdec1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 -- 172.21.15.23:0/3805256156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 msgr2=0x7efdec19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 0x7efdec19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 -- 172.21.15.23:0/3805256156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 msgr2=0x7efdec1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 0x7efdec1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeb7fe640 1 -- 172.21.15.23:0/3805256156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdec1a1800 con 0x7efdec0fff70 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.860+0000 7efdeaffd640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 0x7efdec19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.863 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efde0000c00 tx=0x7efde002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efde8ff9640 1 -- 172.21.15.23:0/3805256156 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0043020 con 0x7efdec0fff70 2024-03-28T15:52:50.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec1a1a90 con 0x7efdec0fff70 2024-03-28T15:52:50.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efde8ff9640 1 -- 172.21.15.23:0/3805256156 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efde00054e0 con 0x7efdec0fff70 2024-03-28T15:52:50.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efde8ff9640 1 -- 172.21.15.23:0/3805256156 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0003c40 con 0x7efdec0fff70 2024-03-28T15:52:50.864 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.861+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec1a2030 con 0x7efdec0fff70 2024-03-28T15:52:50.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efde8ff9640 1 -- 172.21.15.23:0/3805256156 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efde0046020 con 0x7efdec0fff70 2024-03-28T15:52:50.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 Processor -- start 2024-03-28T15:52:50.870 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 -- start start 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x56347599fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634758ffc20 con 0x56347599a740 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634758a8a40 con 0x56347599b130 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634758fbcc0 con 0x5634759a02a0 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56128/0 (socket says 172.21.15.23:56128) 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 -- 172.21.15.23:0/3622020918 learned_addr learned my addr 172.21.15.23:0/3622020918 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x56347599fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdeaffd640 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efde8ff9640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 0x7efdd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.871 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 -- 172.21.15.23:0/3622020918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 msgr2=0x5634759a2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 -- 172.21.15.23:0/3622020918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 msgr2=0x56347599fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x56347599fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.864+0000 7efdebfff640 1 -- 172.21.15.23:0/3622020918 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634758caf90 con 0x56347599a740 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efde8ff9640 1 -- 172.21.15.23:0/3805256156 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7efde00f0ec0 con 0x7efdec0fff70 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efdebfff640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 0x7efdd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efdebfff640 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efde4009fd0 tx=0x7efde40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef077af46c186f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efdd1ffb640 1 -- 172.21.15.23:0/3622020918 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4004250 con 0x56347599a740 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efdd1ffb640 1 -- 172.21.15.23:0/3622020918 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efde4003db0 con 0x56347599a740 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.865+0000 7efdd1ffb640 1 -- 172.21.15.23:0/3622020918 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde401bcc0 con 0x56347599a740 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- 172.21.15.23:0/3622020918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 msgr2=0x56347599ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7efde4009fd0 tx=0x7efde40118d0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.872 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdebfff640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 0x7efdd40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efde4020000 tx=0x7efde4021000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- 172.21.15.23:0/3622020918 shutdown_connections 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x56347599fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/3622020918 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x56347599ab60 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- 172.21.15.23:0/3622020918 >> 172.21.15.23:0/3622020918 conn(0x563475962a60 msgr2=0x5634759960b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- 172.21.15.23:0/3622020918 shutdown_connections 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- 172.21.15.23:0/3622020918 wait complete. 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- start start 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 Processor -- start 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599b130 0x5634759a5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 0x5634759ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.873 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759a3250 con 0x5634759a02a0 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634758bbe90 con 0x56347599a740 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759a33d0 con 0x56347599b130 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.866+0000 7efdf4a5f740 10 client.0 _ll_register_callbacks cb 0x563475613ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 0x5634759ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:47644/0 (socket says 172.21.15.23:47644) 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 -- 172.21.15.23:0/877161596 learned_addr learned my addr 172.21.15.23:0/877161596 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeaffd640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599b130 0x5634759a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdebfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 0x5634759ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56134/0 (socket says 172.21.15.23:56134) 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 -- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599b130 msgr2=0x5634759a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599b130 0x5634759a5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 -- 172.21.15.23:0/877161596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 msgr2=0x5634759ac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 0x5634759ac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.874 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 -- 172.21.15.23:0/877161596 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634758c89d0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efde0006010 tx=0x7efde002c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a72ea573322864eb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00bd240 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7efde00bb770 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efde4003db0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde00bcdd0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdf4a5f740 10 client.25328 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.867+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5634756142d0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efde4003db0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efde00d60d0 con 0x56347599a740 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 10 client.25328 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 10 client.25328 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 20 client.25328 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.875 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x5634759b13b0 0x5634759b0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 4 client.25328 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 --> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56347590d160 con 0x5634759b13b0 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdeaffd640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x5634759b13b0 0x5634759b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdeaffd640 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x5634759b13b0 0x5634759b0300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=90e1e2fbf874f100 server_cookie=6d29514132877801 in_seq=0 out_seq=0 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.868+0000 7efdcaffd640 10 client.25328 ms_handle_connect on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdcaffd640 1 -- 172.21.15.23:0/877161596 <== mds.1 v2:172.21.15.120:6838/4092312487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56347590d160 con 0x5634759b13b0 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdcaffd640 10 client.25328 handle_command_reply: tid=0 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 client.25328 shutdown 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 msgr2=0x5634759a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efde0006010 tx=0x7efde002c880 comp rx=0 tx=0).stop 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 shutdown_connections 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x5634759b13b0 0x5634759b0300 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5634759a02a0 0x5634759ac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.876 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599b130 0x5634759a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdcaffd640 0 client.25328 ms_handle_reset on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/877161596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599a740 0x5634759a4c20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 >> 172.21.15.23:0/877161596 conn(0x563475962a60 msgr2=0x5634759952f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 shutdown_connections 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 1 -- 172.21.15.23:0/877161596 wait complete. 2024-03-28T15:52:50.877 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.869+0000 7efdf4a5f740 20 client.25328 trim_cache size 0 max 16384 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 Processor -- start 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 -- start start 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 0x563475995bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5634759a02a0 0x563475996110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.892 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b0bd0 con 0x56347599b130 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b1230 con 0x5634759a02a0 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b1890 con 0x56347599a740 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56142/0 (socket says 172.21.15.23:56142) 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 -- 172.21.15.23:0/1684089018 learned_addr learned my addr 172.21.15.23:0/1684089018 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 0x563475995bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdebfff640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5634759a02a0 0x563475996110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 -- 172.21.15.23:0/1684089018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 msgr2=0x563475995bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 0x563475995bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 -- 172.21.15.23:0/1684089018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5634759a02a0 msgr2=0x563475996110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.886+0000 7efdeaffd640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5634759a02a0 0x563475996110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdeaffd640 1 -- 172.21.15.23:0/1684089018 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634756de6b0 con 0x56347599b130 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 0x563475995bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:50.893 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdeaffd640 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x5634759a6100 tx=0x7efddc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7d1676bcaae98b server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdd17fa640 1 -- 172.21.15.23:0/1684089018 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc004490 con 0x56347599b130 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdd17fa640 1 -- 172.21.15.23:0/1684089018 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efddc005500 con 0x56347599b130 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdd17fa640 1 -- 172.21.15.23:0/1684089018 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc004a00 con 0x56347599b130 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 -- 172.21.15.23:0/1684089018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 msgr2=0x5634759ac160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x5634759a6100 tx=0x7efddc005f90 comp rx=0 tx=0).stop 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 -- 172.21.15.23:0/1684089018 shutdown_connections 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5634759a02a0 0x563475996110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599b130 0x5634759ac160 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/1684089018 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x56347599a740 0x563475995bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.887+0000 7efdf4a5f740 1 -- 172.21.15.23:0/1684089018 >> 172.21.15.23:0/1684089018 conn(0x563475962a60 msgr2=0x56347599c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- 172.21.15.23:0/1684089018 shutdown_connections 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- 172.21.15.23:0/1684089018 wait complete. 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- start start 2024-03-28T15:52:50.894 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 Processor -- start 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x5634759abfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b2550 con 0x56347599a740 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b1ef0 con 0x56347599b130 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634759b2bb0 con 0x5634759a02a0 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdf4a5f740 10 client.0 _ll_register_callbacks cb 0x7efde00f4690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdeb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56158/0 (socket says 172.21.15.23:56158) 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdeb7fe640 1 -- 172.21.15.23:0/586221167 learned_addr learned my addr 172.21.15.23:0/586221167 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdeaffd640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x5634759abfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.888+0000 7efdebfff640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.895 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 -- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 msgr2=0x5634759a8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 -- 172.21.15.23:0/586221167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 msgr2=0x5634759abfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x5634759abfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 -- 172.21.15.23:0/586221167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634759a8c20 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdeb7fe640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efde00b9e40 tx=0x7efde0005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd1c1e942dc9f5a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00bcfb0 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7efde00c1360 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efddc005500 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde00bd240 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdf4a5f740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x563475a0d0a0 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efddc005500 con 0x56347599a740 2024-03-28T15:52:50.896 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.889+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7efde00c5780 con 0x56347599a740 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 10 client.15489 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 20 client.15489 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x563475a11590 0x563475a0d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 4 client.15489 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 --> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5634759074b0 con 0x563475a11590 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdeaffd640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x563475a11590 0x563475a0d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdcaffd640 10 client.15489 ms_handle_connect on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdeaffd640 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x563475a11590 0x563475a0d390 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b673e9d60e46871 server_cookie=445b5da30354c6d5 in_seq=0 out_seq=0 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdcaffd640 1 -- 172.21.15.23:0/586221167 <== mds.1 v2:172.21.15.120:6838/4092312487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5634759074b0 con 0x563475a11590 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdcaffd640 10 client.15489 handle_command_reply: tid=0 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.890+0000 7efdf4a5f740 1 client.15489 shutdown 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 msgr2=0x5634759aba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.897 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7efde00b9e40 tx=0x7efde0005e50 comp rx=0 tx=0).stop 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 shutdown_connections 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5634759a02a0 0x5634759a8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56347599b130 0x5634759abfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.120:6838/4092312487,v1:172.21.15.120:6839/4092312487] conn(0x563475a11590 0x563475a0d390 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 --2- 172.21.15.23:0/586221167 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x56347599a740 0x5634759aba90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdcaffd640 0 client.15489 ms_handle_reset on v2:172.21.15.120:6838/4092312487 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 >> 172.21.15.23:0/586221167 conn(0x563475962a60 msgr2=0x563475995bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 shutdown_connections 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 1 -- 172.21.15.23:0/586221167 wait complete. 2024-03-28T15:52:50.898 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf4a5f740 20 client.15489 trim_cache size 0 max 16384 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 1, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24481, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 114.338305557, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 94, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 151.32005365500001 2024-03-28T15:52:50.899 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 msgr2=0x7efdd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 0x7efdd40a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efde4020000 tx=0x7efde4021000 comp rx=0 tx=0).stop 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 msgr2=0x7efdec199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7efde0000c00 tx=0x7efde002c650 comp rx=0 tx=0).stop 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 shutdown_connections 2024-03-28T15:52:50.900 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7efdd40a2800 0x7efdd40a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdec101270 0x7efdec1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7efdec1008d0 0x7efdec19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 --2- 172.21.15.23:0/3805256156 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7efdec0fff70 0x7efdec199be0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 >> 172.21.15.23:0/3805256156 conn(0x7efdec0fbb30 msgr2=0x7efdec0fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 shutdown_connections 2024-03-28T15:52:50.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:50.891+0000 7efdf1e80640 1 -- 172.21.15.23:0/3805256156 wait complete. 2024-03-28T15:52:50.903 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-28T15:52:51.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3631455162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4107390 msgr2=0x7fb7e41097e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.014 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3631455162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4107390 0x7fb7e41097e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009ca0 tx=0x7fb7dc02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3631455162 shutdown_connections 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3631455162 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4107390 0x7fb7e41097e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3631455162 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4104a60 0x7fb7e4106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3631455162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4102130 0x7fb7e4104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3631455162 >> 172.21.15.23:0/3631455162 conn(0x7fb7e40fbb40 msgr2=0x7fb7e40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3631455162 shutdown_connections 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.012+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3631455162 wait complete. 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 Processor -- start 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 -- start start 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.015 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4107390 0x7fb7e419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e410d330 con 0x7fb7e4104a60 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e410d1b0 con 0x7fb7e4107390 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7eb7b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e410d4b0 con 0x7fb7e4102130 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:37588/0 (socket says 172.21.15.23:37588) 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56180/0 (socket says 172.21.15.23:56180) 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 -- 172.21.15.23:0/3597000769 learned_addr learned my addr 172.21.15.23:0/3597000769 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4107390 0x7fb7e419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 -- 172.21.15.23:0/3597000769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4107390 msgr2=0x7fb7e419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4107390 0x7fb7e419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 -- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 msgr2=0x7fb7e4197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.016 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e954e640 1 -- 172.21.15.23:0/3597000769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e419f5c0 con 0x7fb7e4102130 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.013+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7e954e640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000ea60 tx=0x7fb7e000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7da7fc640 1 -- 172.21.15.23:0/3597000769 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e000cde0 con 0x7fb7e4102130 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7da7fc640 1 -- 172.21.15.23:0/3597000769 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e00035f0 con 0x7fb7e4102130 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e419f8b0 con 0x7fb7e4102130 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7da7fc640 1 -- 172.21.15.23:0/3597000769 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0010670 con 0x7fb7e4102130 2024-03-28T15:52:51.017 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.014+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e419fe00 con 0x7fb7e4102130 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.016+0000 7fb7da7fc640 1 -- 172.21.15.23:0/3597000769 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb7e00037a0 con 0x7fb7e4102130 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.016+0000 7fb7ee39f740 1 Processor -- start 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.016+0000 7fb7ee39f740 1 -- start start 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c5fb50 0x55ed71c5ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c656b0 0x55ed71c67b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.024 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71bc0c20 con 0x55ed71c60540 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71b69a40 con 0x55ed71c5fb50 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71bbccc0 con 0x55ed71c656b0 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56192/0 (socket says 172.21.15.23:56192) 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 -- 172.21.15.23:0/124880788 learned_addr learned my addr 172.21.15.23:0/124880788 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c656b0 0x55ed71c67b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c5fb50 0x55ed71c5ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 -- 172.21.15.23:0/124880788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c656b0 msgr2=0x55ed71c67b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c656b0 0x55ed71c67b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.025 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7da7fc640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 -- 172.21.15.23:0/124880788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c5fb50 msgr2=0x55ed71c5ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c5fb50 0x55ed71c5ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 -- 172.21.15.23:0/124880788 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed71b8bf90 con 0x55ed71c60540 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7e954e640 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb7e009c240 tx=0x7fb7e00d7500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310e978425c10dc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7da7fc640 1 -- 172.21.15.23:0/3597000769 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fb7e00cf140 con 0x7fb7e4102130 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.017+0000 7fb7cf7fe640 1 -- 172.21.15.23:0/124880788 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e009b4c0 con 0x55ed71c60540 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7cf7fe640 1 -- 172.21.15.23:0/124880788 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e009c080 con 0x55ed71c60540 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7cf7fe640 1 -- 172.21.15.23:0/124880788 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e009b050 con 0x55ed71c60540 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 -- 172.21.15.23:0/124880788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 msgr2=0x55ed71c65040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005e30 tx=0x7fb7dc02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb7e009c240 tx=0x7fb7e00d7500 comp rx=0 tx=0).stop 2024-03-28T15:52:51.026 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 -- 172.21.15.23:0/124880788 shutdown_connections 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c656b0 0x55ed71c67b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c65040 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/124880788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c5fb50 0x55ed71c5ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 -- 172.21.15.23:0/124880788 >> 172.21.15.23:0/124880788 conn(0x55ed71c23a60 msgr2=0x55ed71c570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 -- 172.21.15.23:0/124880788 shutdown_connections 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.018+0000 7fb7ee39f740 1 -- 172.21.15.23:0/124880788 wait complete. 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 -- start start 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 Processor -- start 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c715b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c68660 con 0x55ed71c60540 2024-03-28T15:52:51.027 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71b7ce90 con 0x55ed71c656b0 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c687e0 con 0x55ed71c5fb50 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7ee39f740 10 client.0 _ll_register_callbacks cb 0x55ed718d4ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56204/0 (socket says 172.21.15.23:56204) 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/541727962 learned_addr learned my addr 172.21.15.23:0/541727962 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/541727962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 msgr2=0x55ed71c6a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 msgr2=0x55ed71c715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.019+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/541727962 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed71b899d0 con 0x55ed71c60540 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7e954e640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:51.028 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4012d20 tx=0x7fb7d400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=361340487b766939 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4012110 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed71bce160 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fb7d4004050 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d4025050 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 10 client.15501 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed718d52d0 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ed71bce160 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb7d4010430 con 0x55ed71c60540 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 10 client.15501 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 10 client.15501 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 20 client.15501 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71c76900 0x55ed71c75980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.029 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 4 client.15501 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.020+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed71bc84b0 con 0x55ed71c76900 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.021+0000 7fb7e954e640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71c76900 0x55ed71c75980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.021+0000 7fb7e954e640 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71c76900 0x55ed71c75980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=491498b0aa17c614 server_cookie=88c78167a0e5b643 in_seq=0 out_seq=0 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.021+0000 7fb7ccff9640 10 client.15501 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ccff9640 1 -- 172.21.15.23:0/541727962 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed71bc84b0 con 0x55ed71c76900 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ccff9640 10 client.15501 handle_command_reply: tid=0 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 client.15501 shutdown 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 msgr2=0x55ed71c6a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4012d20 tx=0x7fb7d400f790 comp rx=0 tx=0).stop 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 shutdown_connections 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71c76900 0x55ed71c75980 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.030 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6a570 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ccff9640 0 client.15501 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/541727962 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 >> 172.21.15.23:0/541727962 conn(0x55ed71c23a60 msgr2=0x55ed71c562f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 shutdown_connections 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 1 -- 172.21.15.23:0/541727962 wait complete. 2024-03-28T15:52:51.031 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.022+0000 7fb7ee39f740 20 client.15501 trim_cache size 0 max 16384 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 Processor -- start 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 -- start start 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c60540 0x55ed71c71570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c5a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c76120 con 0x55ed71c5fb50 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c76780 con 0x55ed71c656b0 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c76de0 con 0x55ed71c60540 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56206/0 (socket says 172.21.15.23:56206) 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c60540 0x55ed71c71570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 -- 172.21.15.23:0/1627487820 learned_addr learned my addr 172.21.15.23:0/1627487820 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c5a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 -- 172.21.15.23:0/1627487820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c60540 msgr2=0x55ed71c71570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c60540 0x55ed71c71570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 -- 172.21.15.23:0/1627487820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 msgr2=0x55ed71c5a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c5a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 -- 172.21.15.23:0/1627487820 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed7199f6b0 con 0x55ed71c5fb50 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c5a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7e954e640 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00cc8a0 tx=0x7fb7e0009490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4eb7b6d6c61b5d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ceffd640 1 -- 172.21.15.23:0/1627487820 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e009b230 con 0x55ed71c5fb50 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ceffd640 1 -- 172.21.15.23:0/1627487820 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e00b4070 con 0x55ed71c5fb50 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.039+0000 7fb7ceffd640 1 -- 172.21.15.23:0/1627487820 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e009b4c0 con 0x55ed71c5fb50 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- 172.21.15.23:0/1627487820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 msgr2=0x55ed71c6a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb7e00cc8a0 tx=0x7fb7e0009490 comp rx=0 tx=0).stop 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- 172.21.15.23:0/1627487820 shutdown_connections 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c5a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c60540 0x55ed71c71570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/1627487820 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c5fb50 0x55ed71c6a350 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- 172.21.15.23:0/1627487820 >> 172.21.15.23:0/1627487820 conn(0x55ed71c23a60 msgr2=0x55ed71c69c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- 172.21.15.23:0/1627487820 shutdown_connections 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- 172.21.15.23:0/1627487820 wait complete. 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 -- start start 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.040+0000 7fb7ee39f740 1 Processor -- start 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c70860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c77aa0 con 0x55ed71c60540 2024-03-28T15:52:51.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c77440 con 0x55ed71c656b0 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed71c78100 con 0x55ed71c5fb50 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ee39f740 10 client.0 _ll_register_callbacks cb 0x7fb7d4009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56214/0 (socket says 172.21.15.23:56214) 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e954e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:37624/0 (socket says 172.21.15.23:37624) 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/714631610 learned_addr learned my addr 172.21.15.23:0/714631610 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e9d4f640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c70860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/714631610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 msgr2=0x55ed71c6d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 msgr2=0x55ed71c70860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c70860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 -- 172.21.15.23:0/714631610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed71c70e50 con 0x55ed71c60540 2024-03-28T15:52:51.048 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e954e640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7e8d4d640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40059f0 tx=0x7fb7d400fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb577d6b0c7b62ff server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.041+0000 7fb7ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4022040 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fb7d4010430 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7e00b4070 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d403a050 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed71cd1fe0 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7e00b4070 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 28) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb7d40035f0 con 0x55ed71c60540 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 10 client.15507 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 20 client.15507 populate_metadata read hostname 'smithi023' 2024-03-28T15:52:51.049 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71cd6660 0x55ed71cd2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 4 client.15507 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ed71b469a0 con 0x55ed71cd6660 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.042+0000 7fb7e954e640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71cd6660 0x55ed71cd2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7e954e640 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71cd6660 0x55ed71cd2210 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1aa44cc4c6c75ced server_cookie=c6e77eac74c03fe1 in_seq=0 out_seq=0 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ccff9640 10 client.15507 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ccff9640 1 -- 172.21.15.23:0/714631610 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ed71b469a0 con 0x55ed71cd6660 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ccff9640 10 client.15507 handle_command_reply: tid=0 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 client.15507 shutdown 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 msgr2=0x55ed71c6d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb7d40059f0 tx=0x7fb7d400fd10 comp rx=0 tx=0).stop 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 shutdown_connections 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ed71cd6660 0x55ed71cd2210 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.050 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ed71c656b0 0x55ed71c70860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ed71c60540 0x55ed71c6d840 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ccff9640 0 client.15507 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 --2- 172.21.15.23:0/714631610 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ed71c5fb50 0x55ed71c6d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 >> 172.21.15.23:0/714631610 conn(0x55ed71c23a60 msgr2=0x55ed71c68bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 shutdown_connections 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 1 -- 172.21.15.23:0/714631610 wait complete. 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.043+0000 7fb7ee39f740 20 client.15507 trim_cache size 0 max 16384 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stdout: "cluster_fsid": "a96af96e-ed19-11ee-95d0-87774f69a715", 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stdout: "whoami": 0, 2024-03-28T15:52:51.051 INFO:teuthology.orchestra.run.smithi023.stdout: "id": 24494, 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "want_state": "up:active", 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "state": "up:active", 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "fs_name": "cephfs", 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "rank_uptime": 138.62497672000001, 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "mdsmap_epoch": 28, 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch": 94, 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_epoch_barrier": 90, 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout: "uptime": 143.08296290000001 2024-03-28T15:52:51.052 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 msgr2=0x7fb7d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc005e30 tx=0x7fb7dc02e040 comp rx=0 tx=0).stop 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 msgr2=0x7fb7e41979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000ea60 tx=0x7fb7e000ef20 comp rx=0 tx=0).stop 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 shutdown_connections 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb7d00a2800 0x7fb7d00a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7e4107390 0x7fb7e419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb7e4104a60 0x7fb7e4197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 --2- 172.21.15.23:0/3597000769 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb7e4102130 0x7fb7e41979a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:51.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 >> 172.21.15.23:0/3597000769 conn(0x7fb7e40fbb40 msgr2=0x7fb7e4102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:51.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 shutdown_connections 2024-03-28T15:52:51.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:51.044+0000 7fb7eb7b2640 1 -- 172.21.15.23:0/3597000769 wait complete. 2024-03-28T15:52:51.055 INFO:teuthology.run_tasks:Running task kclient... 2024-03-28T15:52:51.068 INFO:tasks.kclient:Mounting kernel clients... 2024-03-28T15:52:51.068 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-28T15:52:51.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:51.068 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T15:52:51.082 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-28T15:52:51.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:51.083 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-28T15:52:51.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:51.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-28T15:52:51.251 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-28T15:52:51.251 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-28T15:52:51.251 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T15:52:51.251 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-28T15:52:51.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:51 smithi023 ceph-mon[32448]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:52:51.526 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T15:52:51.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:51 smithi117 ceph-mon[36002]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:52:51.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:51 smithi120 ceph-mon[36826]: pgmap v362: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-03-28T15:52:53.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 -- 172.21.15.23:0/111149211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 msgr2=0x7f9de01014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 --2- 172.21.15.23:0/111149211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de01014f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9dd4009990 tx=0x7f9dd4009ca0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 -- 172.21.15.23:0/111149211 shutdown_connections 2024-03-28T15:52:53.237 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 --2- 172.21.15.23:0/111149211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de0101a30 0x7f9de010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 --2- 172.21.15.23:0/111149211 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de01070d0 0x7f9de01074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 --2- 172.21.15.23:0/111149211 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de01014f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 -- 172.21.15.23:0/111149211 >> 172.21.15.23:0/111149211 conn(0x7f9de0077fc0 msgr2=0x7f9de00783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 -- 172.21.15.23:0/111149211 shutdown_connections 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.235+0000 7f9de4bea700 1 -- 172.21.15.23:0/111149211 wait complete. 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 Processor -- start 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 -- start start 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 0x7f9de0199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de01070d0 0x7f9de01a47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0111330 con 0x7f9de0101090 2024-03-28T15:52:53.238 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de01111b0 con 0x7f9de01070d0 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9de4bea700 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de01114b0 con 0x7f9de0101a30 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dde59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dde59c700 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56232/0 (socket says 172.21.15.23:56232) 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dde59c700 1 -- 172.21.15.23:0/3508501686 learned_addr learned my addr 172.21.15.23:0/3508501686 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 0x7f9de0199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dded9d700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de01070d0 0x7f9de01a47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 -- 172.21.15.23:0/3508501686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de01070d0 msgr2=0x7f9de01a47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de01070d0 0x7f9de01a47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 -- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 msgr2=0x7f9de0199770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dddd9b700 1 -- 172.21.15.23:0/3508501686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de01a4d30 con 0x7f9de0101a30 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.236+0000 7f9dde59c700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:53.239 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9dddd9b700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 0x7f9de0199cb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40059d0 tx=0x7f9dd400bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:53.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4027070 con 0x7f9de0101a30 2024-03-28T15:52:53.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de01a4f00 con 0x7f9de0101a30 2024-03-28T15:52:53.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dd4003d70 con 0x7f9de0101a30 2024-03-28T15:52:53.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd4022e30 con 0x7f9de0101a30 2024-03-28T15:52:53.240 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.237+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de01a52f0 con 0x7f9de0101a30 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.238+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0068e50 con 0x7f9de0101a30 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.239+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9dd402a070 con 0x7f9de0101a30 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.239+0000 7f9dcf7fe700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 0x7f9dc80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.240+0000 7f9dde59c700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 0x7f9dc80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.240+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f9dd40cfc60 con 0x7f9de0101a30 2024-03-28T15:52:53.241 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.240+0000 7f9dde59c700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 0x7f9dc80a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0011550 tx=0x7f9dd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:53.243 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.243+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9dd4013070 con 0x7f9de0101a30 2024-03-28T15:52:53.497 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.496+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9de002dbf0 con 0x7f9de0101a30 2024-03-28T15:52:53.497 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:53 smithi023 ceph-mon[32448]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:53.498 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.497+0000 7f9dcf7fe700 1 -- 172.21.15.23:0/3508501686 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v28) v1 ==== 53+0+88 (secure 0 0 0) 0x7f9dd409c940 con 0x7f9de0101a30 2024-03-28T15:52:53.498 INFO:teuthology.orchestra.run.smithi023.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-28T15:52:53.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 msgr2=0x7f9dc80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 0x7f9dc80a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9dd0011550 tx=0x7f9dd0009260 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 msgr2=0x7f9de0199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 0x7f9de0199cb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9dd40059d0 tx=0x7f9dd400bab0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 shutdown_connections 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9dc80a2890 0x7f9dc80a4d50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9de01070d0 0x7f9de01a47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9de0101090 0x7f9de0199770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 --2- 172.21.15.23:0/3508501686 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9de0101a30 0x7f9de0199cb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 >> 172.21.15.23:0/3508501686 conn(0x7f9de0077fc0 msgr2=0x7f9de00ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 shutdown_connections 2024-03-28T15:52:53.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:53.499+0000 7f9de4bea700 1 -- 172.21.15.23:0/3508501686 wait complete. 2024-03-28T15:52:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:53 smithi117 ceph-mon[36002]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:53 smithi120 ceph-mon[36826]: pgmap v363: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi023.front.sepia.ceph.com 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-28T15:52:54.144 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-28T15:52:54.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.145 DEBUG:teuthology.orchestra.run.smithi023:> ip addr 2024-03-28T15:52:54.160 INFO:teuthology.orchestra.run.smithi023.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-28T15:52:54.160 INFO:teuthology.orchestra.run.smithi023.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: inet 127.0.0.1/8 scope host lo 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 ::1/128 scope host 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:ae brd ff:ff:ff:ff:ff:ff 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f0 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:6b:fd:af brd ff:ff:ff:ff:ff:ff 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp7s0f1 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:42 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f0 2024-03-28T15:52:54.161 INFO:teuthology.orchestra.run.smithi023.stdout: inet 172.21.15.23/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft 42060sec preferred_lft 42060sec 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout: inet6 fe80::ec4:7aff:fe59:ce42/64 scope link noprefixroute 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout: valid_lft forever preferred_lft forever 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout: link/ether 0c:c4:7a:59:ce:43 brd ff:ff:ff:ff:ff:ff 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run.smithi023.stdout: altname enp1s0f1 2024-03-28T15:52:54.162 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-28T15:52:54.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.162 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:52:54.162 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:52:54.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add name ceph-brx type bridge 2024-03-28T15:52:54.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr flush dev ceph-brx 2024-03-28T15:52:54.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set ceph-brx up 2024-03-28T15:52:54.163 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-28T15:52:54.163 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:52:54.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.271 DEBUG:teuthology.orchestra.run.smithi023:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-28T15:52:54.298 INFO:teuthology.orchestra.run.smithi023.stdout:1 2024-03-28T15:52:54.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.299 DEBUG:teuthology.orchestra.run.smithi023:> route 2024-03-28T15:52:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:Kernel IP routing table 2024-03-28T15:52:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-28T15:52:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-28T15:52:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-28T15:52:54.356 INFO:teuthology.orchestra.run.smithi023.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-28T15:52:54.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-28T15:52:54.357 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:52:54.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.452 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T15:52:54.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.467 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list-id 2024-03-28T15:52:54.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.522 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:52:54.522 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:52:54.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T15:52:54.522 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-28T15:52:54.523 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:52:54.607 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-28T15:52:54.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.607 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:52:54.607 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:52:54.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-28T15:52:54.607 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-28T15:52:54.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-28T15:52:54.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-28T15:52:54.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-28T15:52:54.608 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:52:54.676 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3508501686' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:52:54.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.694 DEBUG:teuthology.orchestra.run.smithi023:> (cd / && exec bash -c ' 2024-03-28T15:52:54.694 DEBUG:teuthology.orchestra.run.smithi023:> set -e 2024-03-28T15:52:54.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set brx.0 up 2024-03-28T15:52:54.695 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link set dev brx.0 master ceph-brx 2024-03-28T15:52:54.695 DEBUG:teuthology.orchestra.run.smithi023:> ') 2024-03-28T15:52:54.736 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:52:54.736 DEBUG:teuthology.orchestra.run.smithi023:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-28T15:52:54.790 INFO:teuthology.orchestra.run.smithi023.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-28T15:52:54.790 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-28T15:52:54.790 DEBUG:teuthology.orchestra.run.smithi023:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-28T15:52:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:54 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3508501686' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:52:54.838 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:54 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3508501686' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-28T15:52:54.843 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-28T15:52:54.843 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-28T15:52:54.844 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/221c8d1e-ed4e-48f2-bb79-dba7ddaab0a5 2024-03-28T15:52:54.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:54.844 DEBUG:teuthology.orchestra.run.smithi023:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/221c8d1e-ed4e-48f2-bb79-dba7ddaab0a5 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T15:52:54.928 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T15:52:54.928 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T15:52:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2024-03-28T15:52:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2024-03-28T15:52:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: resolved to: "172.21.15.23:3300,172.21.15.117:3300,172.21.15.120:3300" 2024-03-28T15:52:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: trying mount with new device syntax: 0@a96af96e-ed19-11ee-95d0-87774f69a715.cephfs=/volumes/qa/sv_1/221c8d1e-ed4e-48f2-bb79-dba7ddaab0a5 2024-03-28T15:52:55.071 INFO:teuthology.orchestra.run.smithi023.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.23:3300/172.21.15.117:3300/172.21.15.120:3300" will pass to kernel 2024-03-28T15:52:55.072 INFO:teuthology.orchestra.run.smithi023.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-28T15:52:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: You just mounted a file system that supports labels which does not 2024-03-28T15:52:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-28T15:52:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: applications will generate AVC messages and not be allowed access to 2024-03-28T15:52:55.072 INFO:teuthology.orchestra.run.smithi023.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-28T15:52:55.074 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-28T15:52:55.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:55.074 DEBUG:teuthology.orchestra.run.smithi023:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-28T15:52:55.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:55.147 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T15:52:55.203 INFO:teuthology.orchestra.run.smithi023.stdout:client15516 2024-03-28T15:52:55.204 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:52:55.204 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T15:52:55.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:55.260 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T15:52:55.315 INFO:teuthology.orchestra.run.smithi023.stdout:client15516 2024-03-28T15:52:55.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:55.316 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-28T15:52:55.371 INFO:teuthology.orchestra.run.smithi023.stdout:client15516 2024-03-28T15:52:55.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T15:52:55.372 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/a96af96e-ed19-11ee-95d0-87774f69a715.client15516/status') 2024-03-28T15:52:55.439 INFO:teuthology.orchestra.run.smithi023.stdout:instance: client.15516 (3)192.168.0.1:0/1596125135 2024-03-28T15:52:55.439 INFO:teuthology.orchestra.run.smithi023.stdout:blocklisted: false 2024-03-28T15:52:55.439 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records in 2024-03-28T15:52:55.439 INFO:teuthology.orchestra.run.smithi023.stderr:0+1 records out 2024-03-28T15:52:55.439 INFO:teuthology.orchestra.run.smithi023.stderr:70 bytes copied, 2.3342e-05 s, 3.0 MB/s 2024-03-28T15:52:55.440 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T15:52:55.450 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T15:52:55.450 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:52:55.451 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/1859528536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f0102140 msgr2=0x7fb0f0104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/1859528536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f0102140 0x7fb0f0104530 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb0e800fb60 tx=0x7fb0e802fcf0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/1859528536 shutdown_connections 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/1859528536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f01073a0 0x7fb0f01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/1859528536 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0104a70 0x7fb0f0106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/1859528536 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f0102140 0x7fb0f0104530 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/1859528536 >> 172.21.15.23:0/1859528536 conn(0x7fb0f00fbb30 msgr2=0x7fb0f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/1859528536 shutdown_connections 2024-03-28T15:52:55.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.585+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/1859528536 wait complete. 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 Processor -- start 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 -- start start 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f0104a70 0x7fb0f0195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f01073a0 0x7fb0f019ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010b180 con 0x7fb0f0102140 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010b000 con 0x7fb0f0104a70 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f7a4b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f010b300 con 0x7fb0f01073a0 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f57e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f57e7640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56246/0 (socket says 172.21.15.23:56246) 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f57e7640 1 -- 172.21.15.23:0/2809601127 learned_addr learned my addr 172.21.15.23:0/2809601127 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f5fe8640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f01073a0 0x7fb0f019ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f4fe6640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f0104a70 0x7fb0f0195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:55.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.586+0000 7fb0f57e7640 1 -- 172.21.15.23:0/2809601127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f01073a0 msgr2=0x7fb0f019ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f57e7640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f01073a0 0x7fb0f019ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f57e7640 1 -- 172.21.15.23:0/2809601127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f0104a70 msgr2=0x7fb0f0195c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f57e7640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f0104a70 0x7fb0f0195c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f57e7640 1 -- 172.21.15.23:0/2809601127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f019d330 con 0x7fb0f0102140 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f57e7640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8038c40 tx=0x7fb0e80045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8045020 con 0x7fb0f0102140 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f019d5c0 con 0x7fb0f0102140 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0e8030b70 con 0x7fb0f0102140 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e803d040 con 0x7fb0f0102140 2024-03-28T15:52:55.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.587+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f019db00 con 0x7fb0f0102140 2024-03-28T15:52:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.588+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f00fd620 con 0x7fb0f0102140 2024-03-28T15:52:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.589+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb0e8048020 con 0x7fb0f0102140 2024-03-28T15:52:55.591 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.589+0000 7fb0ee7fc640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 0x7fb0d80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:55.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.590+0000 7fb0f4fe6640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 0x7fb0d80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:55.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.590+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fb0e80f47d0 con 0x7fb0f0102140 2024-03-28T15:52:55.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.590+0000 7fb0f4fe6640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 0x7fb0d80a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0196c20 tx=0x7fb0e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:55.594 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.594+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb0e804f020 con 0x7fb0f0102140 2024-03-28T15:52:55.818 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:55.818+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7fb0f002d070 con 0x7fb0f0102140 2024-03-28T15:52:55.838 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:55 smithi117 ceph-mon[36002]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:55.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:55 smithi120 ceph-mon[36826]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:55.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:55 smithi023 ceph-mon[32448]: pgmap v364: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-28T15:52:56.473 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.473+0000 7fb0ee7fc640 1 -- 172.21.15.23:0/2809601127 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v29) v1 ==== 117+0+0 (secure 0 0 0) 0x7fb0e80c1480 con 0x7fb0f0102140 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.475+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 msgr2=0x7fb0d80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.475+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 0x7fb0d80a4d60 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0f0196c20 tx=0x7fb0e4009260 comp rx=0 tx=0).stop 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.475+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 msgr2=0x7fb0f0195710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.475+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8038c40 tx=0x7fb0e80045a0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 shutdown_connections 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fb0d80a28a0 0x7fb0d80a4d60 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb0f01073a0 0x7fb0f019ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0f0104a70 0x7fb0f0195c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.476 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 --2- 172.21.15.23:0/2809601127 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb0f0102140 0x7fb0f0195710 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 >> 172.21.15.23:0/2809601127 conn(0x7fb0f00fbb30 msgr2=0x7fb0f0102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 shutdown_connections 2024-03-28T15:52:56.477 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.476+0000 7fb0f7a4b640 1 -- 172.21.15.23:0/2809601127 wait complete. 2024-03-28T15:52:56.488 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-28T15:52:56.499 INFO:teuthology.run_tasks:Running task exec... 2024-03-28T15:52:56.561 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T15:52:56.561 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:52:56.562 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.664+0000 7fbdc4cde640 1 -- 172.21.15.23:0/1368368222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01028e0 msgr2=0x7fbdc0102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.664+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/1368368222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01028e0 0x7fbdc0102d60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbdb4009900 tx=0x7fbdb402bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- 172.21.15.23:0/1368368222 shutdown_connections 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/1368368222 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01032a0 0x7fbdc0109b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/1368368222 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01028e0 0x7fbdc0102d60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/1368368222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01016f0 0x7fbdc0101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- 172.21.15.23:0/1368368222 >> 172.21.15.23:0/1368368222 conn(0x7fbdc00fcc60 msgr2=0x7fbdc00ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- 172.21.15.23:0/1368368222 shutdown_connections 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- 172.21.15.23:0/1368368222 wait complete. 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 Processor -- start 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- start start 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 0x7fbdc019a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01032a0 0x7fbdc01a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc010d370 con 0x7fbdc01032a0 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc010d1f0 con 0x7fbdc01028e0 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.665+0000 7fbdc4cde640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc010d4f0 con 0x7fbdc01016f0 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbdd9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 0x7fbdc019a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:56.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:37684/0 (socket says 172.21.15.23:37684) 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 -- 172.21.15.23:0/2097055370 learned_addr learned my addr 172.21.15.23:0/2097055370 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbed9d640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01032a0 0x7fbdc01a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 -- 172.21.15.23:0/2097055370 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 msgr2=0x7fbdc019a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 0x7fbdc019a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 -- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01032a0 msgr2=0x7fbdc01a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01032a0 0x7fbdc01a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 -- 172.21.15.23:0/2097055370 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc01a1730 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbdd9b640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 0x7fbdc019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdbe59c640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e9c0 tx=0x7fbdb800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800cd80 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbdb80035f0 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdc01a1a20 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdb8010690 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.666+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc01a1f90 con 0x7fbdc01016f0 2024-03-28T15:52:56.669 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.667+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdc01a2220 con 0x7fbdc01016f0 2024-03-28T15:52:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.668+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbdb80037a0 con 0x7fbdc01016f0 2024-03-28T15:52:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.669+0000 7fbdaf7fe640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 0x7fbda80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:56.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.669+0000 7fbdbdd9b640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 0x7fbda80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.669+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7fbdb80cf2b0 con 0x7fbdc01016f0 2024-03-28T15:52:56.671 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.670+0000 7fbdbdd9b640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 0x7fbda80a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402c7b0 tx=0x7fbdb4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:56.672 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.672+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbdb809bf90 con 0x7fbdc01016f0 2024-03-28T15:52:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:56 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T15:52:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:56 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T15:52:56.907 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:56.907+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fbdc01a23f0 con 0x7fbdc01016f0 2024-03-28T15:52:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:56 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-28T15:52:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.490+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fbdb8097f00 con 0x7fbdc01016f0 2024-03-28T15:52:57.491 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.491+0000 7fbdaf7fe640 1 -- 172.21.15.23:0/2097055370 <== mon.1 v2:172.21.15.120:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7fbdb8024840 con 0x7fbdc01016f0 2024-03-28T15:52:57.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 msgr2=0x7fbda80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.494 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 0x7fbda80a4de0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbdb402c7b0 tx=0x7fbdb4005fd0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 msgr2=0x7fbdc0199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800e9c0 tx=0x7fbdb800ee80 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 shutdown_connections 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7fbda80a2920 0x7fbda80a4de0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fbdc01032a0 0x7fbdc01a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc01028e0 0x7fbdc019a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 --2- 172.21.15.23:0/2097055370 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fbdc01016f0 0x7fbdc0199b10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 >> 172.21.15.23:0/2097055370 conn(0x7fbdc00fcc60 msgr2=0x7fbdc00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 shutdown_connections 2024-03-28T15:52:57.495 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.494+0000 7fbdc4cde640 1 -- 172.21.15.23:0/2097055370 wait complete. 2024-03-28T15:52:57.510 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-28T15:52:57.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 -- 172.21.15.23:0/4275369187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 msgr2=0x7f4858104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.624 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 --2- 172.21.15.23:0/4275369187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858104530 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f4848009900 tx=0x7f484802bc30 comp rx=0 tx=0).stop 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 -- 172.21.15.23:0/4275369187 shutdown_connections 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 --2- 172.21.15.23:0/4275369187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48581073a0 0x7f48581097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 --2- 172.21.15.23:0/4275369187 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4858104a70 0x7f4858106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 --2- 172.21.15.23:0/4275369187 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858104530 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 -- 172.21.15.23:0/4275369187 >> 172.21.15.23:0/4275369187 conn(0x7f48580fbb30 msgr2=0x7f48580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 -- 172.21.15.23:0/4275369187 shutdown_connections 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 -- 172.21.15.23:0/4275369187 wait complete. 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.620+0000 7f485dd87640 1 Processor -- start 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 -- start start 2024-03-28T15:52:57.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 0x7f48581000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 0x7f4858100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485810d360 con 0x7f4858102140 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485810d1e0 con 0x7f4858104a70 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f485dd87640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485810d4e0 con 0x7f48581073a0 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56304/0 (socket says 172.21.15.23:56304) 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f4856ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 0x7f48581000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 -- 172.21.15.23:0/1528519658 learned_addr learned my addr 172.21.15.23:0/1528519658 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:57.626 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f4857fff640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 0x7f4858100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 -- 172.21.15.23:0/1528519658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 msgr2=0x7f4858100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 0x7f4858100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 -- 172.21.15.23:0/1528519658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 msgr2=0x7f48581000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 0x7f48581000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f48577fe640 1 -- 172.21.15.23:0/1528519658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4858100ec0 con 0x7f4858102140 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f4856ffd640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 0x7f48581000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.621+0000 7f4857fff640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 0x7f4858100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f48577fe640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4848009230 tx=0x7f4848005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848041070 con 0x7f4858102140 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48581a1a30 con 0x7f4858102140 2024-03-28T15:52:57.627 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48480044d0 con 0x7f4858102140 2024-03-28T15:52:57.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848034440 con 0x7f4858102140 2024-03-28T15:52:57.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.622+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48581a1fd0 con 0x7f4858102140 2024-03-28T15:52:57.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.623+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48580fd620 con 0x7f4858102140 2024-03-28T15:52:57.628 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.624+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f4848044070 con 0x7f4858102140 2024-03-28T15:52:57.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.625+0000 7f4854ff9640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 0x7f48400a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.625+0000 7f4856ffd640 1 -- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 msgr2=0x7f48400a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:57.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.625+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f48480f0a90 con 0x7f4858102140 2024-03-28T15:52:57.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.625+0000 7f4856ffd640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 0x7f48400a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:52:57.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.629+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f48480bd0f0 con 0x7f4858102140 2024-03-28T15:52:57.814 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.814+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f485802d070 con 0x7f4858102140 2024-03-28T15:52:57.819 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.819+0000 7f4854ff9640 1 -- 172.21.15.23:0/1528519658 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f48480c1190 con 0x7f4858102140 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: ignoring --setuser ceph since I am not root 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: ignoring --setgroup ceph since I am not root 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mgr[37387]: -- 172.21.15.117:0/2650369608 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564d1439e5a0 con 0x564d14372c00 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mon[36002]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mon[36002]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:52:57.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2097055370' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:57 smithi117 ceph-mon[36002]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 msgr2=0x7f48400a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:52:57.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 0x7f48400a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.822 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 msgr2=0x7f4858101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f4848009230 tx=0x7f4848005d50 comp rx=0 tx=0).stop 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 shutdown_connections 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f48400a28f0 0x7f48400a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f48581073a0 0x7f4858100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4858104a70 0x7f48581000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 --2- 172.21.15.23:0/1528519658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4858102140 0x7f4858101ab0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 >> 172.21.15.23:0/1528519658 conn(0x7f48580fbb30 msgr2=0x7f48581029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 shutdown_connections 2024-03-28T15:52:57.823 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.822+0000 7f485dd87640 1 -- 172.21.15.23:0/1528519658 wait complete. 2024-03-28T15:52:57.836 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-mon[32448]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:57.836 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T15:52:57.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-mon[32448]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:52:57.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2097055370' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.837 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-mon[32448]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.837 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setuser ceph since I am not root 2024-03-28T15:52:57.837 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:57 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: ignoring --setgroup ceph since I am not root 2024-03-28T15:52:57.838 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-mon[36826]: pgmap v365: 129 pgs: 129 active+clean; 641 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2809601127' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-mon[36826]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2097055370' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-mon[36826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-28T15:52:57.839 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: ignoring --setuser ceph since I am not root 2024-03-28T15:52:57.840 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: ignoring --setgroup ceph since I am not root 2024-03-28T15:52:57.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 -- 172.21.15.23:0/2011153207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 msgr2=0x7f9b341097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 --2- 172.21.15.23:0/2011153207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b341097f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009ca0 tx=0x7f9b2c02c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 -- 172.21.15.23:0/2011153207 shutdown_connections 2024-03-28T15:52:57.946 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 --2- 172.21.15.23:0/2011153207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b341097f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 --2- 172.21.15.23:0/2011153207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34104a70 0x7f9b34106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 --2- 172.21.15.23:0/2011153207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34102140 0x7f9b34104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 -- 172.21.15.23:0/2011153207 >> 172.21.15.23:0/2011153207 conn(0x7f9b340fbb30 msgr2=0x7f9b340fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 -- 172.21.15.23:0/2011153207 shutdown_connections 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 -- 172.21.15.23:0/2011153207 wait complete. 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.944+0000 7f9b39268640 1 Processor -- start 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 -- start start 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34102140 0x7f9b341978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 0x7f9b34197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3410d360 con 0x7f9b341073a0 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3410d1e0 con 0x7f9b34104a70 2024-03-28T15:52:57.947 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b39268640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3410d4e0 con 0x7f9b34102140 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b32ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34102140 0x7f9b341978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56322/0 (socket says 172.21.15.23:56322) 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 -- 172.21.15.23:0/3108694564 learned_addr learned my addr 172.21.15.23:0/3108694564 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b327fc640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 0x7f9b34197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 -- 172.21.15.23:0/3108694564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34102140 msgr2=0x7f9b341978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34102140 0x7f9b341978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 -- 172.21.15.23:0/3108694564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 msgr2=0x7f9b34197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 0x7f9b34197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b337fe640 1 -- 172.21.15.23:0/3108694564 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3419f4d0 con 0x7f9b341073a0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.945+0000 7f9b327fc640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 0x7f9b34197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b337fe640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c037040 tx=0x7f9b2c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c041020 con 0x7f9b341073a0 2024-03-28T15:52:57.948 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b2c034cd0 con 0x7f9b341073a0 2024-03-28T15:52:57.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3419f760 con 0x7f9b341073a0 2024-03-28T15:52:57.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c039040 con 0x7f9b341073a0 2024-03-28T15:52:57.949 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.946+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3419fc50 con 0x7f9b341073a0 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.947+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b340fd620 con 0x7f9b341073a0 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.948+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f9b2c044020 con 0x7f9b341073a0 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.949+0000 7f9b1bfff640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 0x7f9b1c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.949+0000 7f9b32ffd640 1 -- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 msgr2=0x7f9b1c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.949+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f9b2c0f0d50 con 0x7f9b341073a0 2024-03-28T15:52:57.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.949+0000 7f9b32ffd640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 0x7f9b1c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:52:57.953 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:57.953+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9b2c0f4030 con 0x7f9b341073a0 2024-03-28T15:52:58.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.139+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9b34062310 con 0x7f9b341073a0 2024-03-28T15:52:58.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.143+0000 7f9b1bfff640 1 -- 172.21.15.23:0/3108694564 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9b2c04b020 con 0x7f9b341073a0 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 msgr2=0x7f9b1c0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 0x7f9b1c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 msgr2=0x7f9b3419ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c037040 tx=0x7f9b2c032f40 comp rx=0 tx=0).stop 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 shutdown_connections 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f9b1c0a28f0 0x7f9b1c0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9b341073a0 0x7f9b3419ee60 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9b34104a70 0x7f9b34197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.146 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.145+0000 7f9b39268640 1 --2- 172.21.15.23:0/3108694564 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9b34102140 0x7f9b341978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.146+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 >> 172.21.15.23:0/3108694564 conn(0x7f9b340fbb30 msgr2=0x7f9b34102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:58.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.146+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 shutdown_connections 2024-03-28T15:52:58.147 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.146+0000 7f9b39268640 1 -- 172.21.15.23:0/3108694564 wait complete. 2024-03-28T15:52:58.157 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 -- 172.21.15.23:0/666909886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59781073a0 msgr2=0x7f59781097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 --2- 172.21.15.23:0/666909886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59781073a0 0x7f59781097f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5970009ca0 tx=0x7f597002c0f0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 -- 172.21.15.23:0/666909886 shutdown_connections 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 --2- 172.21.15.23:0/666909886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59781073a0 0x7f59781097f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 --2- 172.21.15.23:0/666909886 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5978104a70 0x7f5978106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 --2- 172.21.15.23:0/666909886 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 -- 172.21.15.23:0/666909886 >> 172.21.15.23:0/666909886 conn(0x7f59780fbb30 msgr2=0x7f59780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 -- 172.21.15.23:0/666909886 shutdown_connections 2024-03-28T15:52:58.298 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.294+0000 7f597ea68640 1 -- 172.21.15.23:0/666909886 wait complete. 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 Processor -- start 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 -- start start 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5978104a70 0x7f59781000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59781073a0 0x7f5978100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597810d360 con 0x7f5978102140 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597810d1e0 con 0x7f5978104a70 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597ea68640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597810d4e0 con 0x7f59781073a0 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56332/0 (socket says 172.21.15.23:56332) 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 -- 172.21.15.23:0/3667061063 learned_addr learned my addr 172.21.15.23:0/3667061063 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:52:58.299 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5978104a70 0x7f59781000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597d005640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59781073a0 0x7f5978100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59781073a0 msgr2=0x7f5978100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f59781073a0 0x7f5978100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5978104a70 msgr2=0x7f59781000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5978104a70 0x7f59781000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978100ec0 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.295+0000 7f597c804640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978101ab0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f596c00ea60 tx=0x7f596c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.296+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c00cde0 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.296+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f596c0035f0 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.296+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c010670 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.296+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59781a1a00 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.296+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59781a1f00 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.297+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59780fd620 con 0x7f5978102140 2024-03-28T15:52:58.300 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.298+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f596c0037a0 con 0x7f5978102140 2024-03-28T15:52:58.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.299+0000 7f5975ffb640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:52:58.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.299+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:58.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.299+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9237+0+0 (secure 0 0 0) 0x7f596c0cf1e0 con 0x7f5978102140 2024-03-28T15:52:58.301 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.299+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-28T15:52:58.303 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.303+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f596c09be30 con 0x7f5978102140 2024-03-28T15:52:58.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:58.122+0000 7ff17fe3d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:52:58.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:58.222+0000 7ff17fe3d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:52:58.486 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:58.159+0000 7fb218878200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:52:58.486 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:58.278+0000 7fb218878200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:52:58.487 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:58.114+0000 7f2395869200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T15:52:58.487 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:58.231+0000 7f2395869200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T15:52:58.499 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.500+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:58.500 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.500+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-28T15:52:58.625 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.625+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5978061ae0 con 0x7f59680a2970 2024-03-28T15:52:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-mon[36002]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T15:52:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-mon[36002]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T15:52:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1528519658' entity='client.admin' 2024-03-28T15:52:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3108694564' entity='client.admin' 2024-03-28T15:52:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-mon[36826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T15:52:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-mon[36826]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T15:52:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1528519658' entity='client.admin' 2024-03-28T15:52:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:52:58 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3108694564' entity='client.admin' 2024-03-28T15:52:58.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.901+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:58.901 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:58.901+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-28T15:52:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-mon[32448]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-28T15:52:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-mon[32448]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-28T15:52:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1528519658' entity='client.admin' 2024-03-28T15:52:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3108694564' entity='client.admin' 2024-03-28T15:52:59.161 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:58.899+0000 7f2395869200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:52:59.162 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:58.993+0000 7f2395869200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:52:59.162 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.160+0000 7f2395869200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:52:59.336 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.002+0000 7fb218878200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:52:59.337 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.097+0000 7fb218878200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:52:59.337 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.239+0000 7fb218878200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:52:59.337 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:58.998+0000 7ff17fe3d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T15:52:59.337 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.094+0000 7ff17fe3d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T15:52:59.337 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.239+0000 7ff17fe3d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T15:52:59.338 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.337+0000 7ff17fe3d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:52:59.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.257+0000 7f2395869200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:52:59.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.341+0000 7f2395869200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:52:59.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.336+0000 7fb218878200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T15:52:59.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.420+0000 7fb218878200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:52:59.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.423+0000 7ff17fe3d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T15:52:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:59.702+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:52:59.702 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:52:59.702+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-28T15:52:59.989 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.655+0000 7f2395869200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:52:59.989 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.791+0000 7f2395869200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:52:59.989 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.891+0000 7f2395869200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:52:59.992 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.693+0000 7ff17fe3d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:52:59.992 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.806+0000 7ff17fe3d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:52:59.992 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.894+0000 7ff17fe3d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:53:00.010 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.687+0000 7fb218878200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T15:53:00.010 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.798+0000 7fb218878200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:00.010 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:52:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:52:59.889+0000 7fb218878200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T15:53:00.321 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:52:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:52:59.987+0000 7f2395869200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:53:00.322 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:00.165+0000 7f2395869200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:53:00.339 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:00.009+0000 7fb218878200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:53:00.339 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:00.203+0000 7fb218878200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:53:00.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:52:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:52:59.992+0000 7ff17fe3d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T15:53:00.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:00.190+0000 7ff17fe3d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T15:53:01.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:01.304+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:53:01.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:01.304+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-28T15:53:01.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:01.222+0000 7f2395869200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:53:01.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:01.368+0000 7f2395869200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:53:01.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:01.308+0000 7fb218878200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:53:01.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:01.454+0000 7fb218878200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:53:01.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:01.309+0000 7ff17fe3d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T15:53:01.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:01.447+0000 7ff17fe3d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T15:53:02.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:01.727+0000 7f2395869200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:53:02.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:01.817+0000 7f2395869200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:53:02.089 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:01.792+0000 7fb218878200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:53:02.089 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:01.882+0000 7fb218878200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:53:02.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:01.874+0000 7ff17fe3d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T15:53:02.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:01.982+0000 7ff17fe3d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T15:53:02.394 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:02.125+0000 7fb218878200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:02.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:02.075+0000 7f2395869200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:02.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:02.319+0000 7f2395869200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:53:02.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:02.254+0000 7ff17fe3d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:02.670 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:02.475+0000 7ff17fe3d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:53:02.731 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:02.393+0000 7fb218878200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T15:53:02.946 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:02.655+0000 7f2395869200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:03.033 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:02.730+0000 7fb218878200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:03.037 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:02.774+0000 7ff17fe3d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T15:53:03.231 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:02.944+0000 7f2395869200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:03.325 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:03.032+0000 7fb218878200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:03.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:03.037+0000 7ff17fe3d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T15:53:03.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:03.283+0000 7ff17fe3d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:53:03.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:03.230+0000 7f2395869200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:53:03.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:03.324+0000 7fb218878200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T15:53:04.420 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:04.147+0000 7ff17fe3d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:53:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:04.507+0000 7f5977fff640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.23:6800/1340727098 2024-03-28T15:53:04.507 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:04.507+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-28T15:53:04.571 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:04.091+0000 7f2395869200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:53:04.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:04.284+0000 7fb218878200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T15:53:05.112 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:04.780+0000 7ff17fe3d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:53:05.112 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:04.884+0000 7ff17fe3d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:53:05.112 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.111+0000 7ff17fe3d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:53:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:05.138+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 20) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f596c097da0 con 0x7f5978102140 2024-03-28T15:53:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:05.138+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 msgr2=0x7f59680a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-28T15:53:05.139 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:05.138+0000 7f5975ffb640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1340727098,v1:172.21.15.23:6801/1340727098] conn(0x7f59680a2970 0x7f59680a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:05.178 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:04.830+0000 7f2395869200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:53:05.178 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:04.929+0000 7f2395869200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:53:05.331 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:04.994+0000 7fb218878200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T15:53:05.331 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:05.093+0000 7fb218878200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T15:53:05.332 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: Active manager daemon x restarted 2024-03-28T15:53:05.332 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: Activating manager daemon x 2024-03-28T15:53:05.332 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T15:53:05.332 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: mgrmap e20: x(active, starting, since 0.0141125s), standbys: z, y 2024-03-28T15:53:05.332 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:53:05.333 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:53:05.334 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:53:05.335 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-mon[36826]: Manager daemon x is now available 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: Active manager daemon x restarted 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: Activating manager daemon x 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: mgrmap e20: x(active, starting, since 0.0141125s), standbys: z, y 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:53:05.422 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:05.423 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:53:05.424 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-mon[32448]: Manager daemon x is now available 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y[37377]: 2024-03-28T15:53:05.205+0000 7f2395869200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: Active manager daemon x restarted 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: Activating manager daemon x 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: osdmap e95: 12 total, 12 up, 12 in 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: mgrmap e20: x(active, starting, since 0.0141125s), standbys: z, y 2024-03-28T15:53:05.531 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T15:53:05.532 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-28T15:53:05.533 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-28T15:53:05.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-28T15:53:05.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-28T15:53:05.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-28T15:53:05.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-28T15:53:05.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:05 smithi117 ceph-mon[36002]: Manager daemon x is now available 2024-03-28T15:53:05.589 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 15:53:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z[38686]: 2024-03-28T15:53:05.362+0000 7fb218878200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T15:53:05.920 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.443+0000 7ff0df7cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.443+0000 7ff0df7cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.443+0000 7ff0df7cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.443+0000 7ff0df7cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.443+0000 7ff0df7cc700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.444+0000 7ff0e27d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.444+0000 7ff0e27d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.444+0000 7ff0e27d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.444+0000 7ff0e27d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:05.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:53:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:53:05.444+0000 7ff0e27d2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:53:06.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.141+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mon.0 v2:172.21.15.23:3300/0 8 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f596c042bd0 con 0x7f5978102140 2024-03-28T15:53:06.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.142+0000 7f5975ffb640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f59680511d0 0x7f59680515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.143 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.142+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f596c009030 con 0x7f59680511d0 2024-03-28T15:53:06.148 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.147+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f59680511d0 0x7f59680515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.151 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.151+0000 7f5977fff640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f59680511d0 0x7f59680515b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5978101820 tx=0x7f5964005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:06.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.226+0000 7f5975ffb640 1 -- 172.21.15.23:0/3667061063 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f596c009030 con 0x7f59680511d0 2024-03-28T15:53:06.227 INFO:teuthology.orchestra.run.smithi023.stdout:Schedule set for path / 2024-03-28T15:53:06.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f59680511d0 msgr2=0x7f59680515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f59680511d0 0x7f59680515b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5978101820 tx=0x7f5964005f90 comp rx=0 tx=0).stop 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 msgr2=0x7f5978101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 --2- 172.21.15.23:0/3667061063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5978102140 0x7f5978101ab0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f596c00ea60 tx=0x7f596c00ef20 comp rx=0 tx=0).stop 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597d005640 1 -- 172.21.15.23:0/3667061063 reap_dead start 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 shutdown_connections 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 >> 172.21.15.23:0/3667061063 conn(0x7f59780fbb30 msgr2=0x7f5978102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 shutdown_connections 2024-03-28T15:53:06.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.228+0000 7f597ea68640 1 -- 172.21.15.23:0/3667061063 wait complete. 2024-03-28T15:53:06.242 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-28T15:53:06.359 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- 172.21.15.23:0/1539658869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 msgr2=0x7f4360101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- 172.21.15.23:0/1539658869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f4360101c50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4350009900 tx=0x7f435002bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- 172.21.15.23:0/1539658869 shutdown_connections 2024-03-28T15:53:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- 172.21.15.23:0/1539658869 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 0x7f4360108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- 172.21.15.23:0/1539658869 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f4360101c50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.360 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- 172.21.15.23:0/1539658869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f4360100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- 172.21.15.23:0/1539658869 >> 172.21.15.23:0/1539658869 conn(0x7f43600fbb50 msgr2=0x7f43600fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- 172.21.15.23:0/1539658869 shutdown_connections 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- 172.21.15.23:0/1539658869 wait complete. 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 Processor -- start 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 -- start start 2024-03-28T15:53:06.361 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.354+0000 7f4368027640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f436019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 0x7f436019ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436010c260 con 0x7f43601017d0 2024-03-28T15:53:06.362 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436010c0e0 con 0x7f43601005e0 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436010c3e0 con 0x7f4360102190 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f436019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4365dc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4365dc3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:58592/0 (socket says 172.21.15.23:58592) 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4365dc3640 1 -- 172.21.15.23:0/2679100804 learned_addr learned my addr 172.21.15.23:0/2679100804 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:06.363 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43665c4640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 0x7f436019ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 -- 172.21.15.23:0/2679100804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 msgr2=0x7f436019ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 0x7f436019ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 -- 172.21.15.23:0/2679100804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 msgr2=0x7f436006a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.364 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 -- 172.21.15.23:0/2679100804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436019f590 con 0x7f43601017d0 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4365dc3640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f43655c2640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f436019c660 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4350009230 tx=0x7f435002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350041070 con 0x7f43601017d0 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436019f820 con 0x7f43601017d0 2024-03-28T15:53:06.365 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4350004450 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350003df0 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.355+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436019fd10 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.356+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4360107900 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.358+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f4350044070 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.358+0000 7f4356ffd640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 0x7f434c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.358+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f43500f04e0 con 0x7f43601017d0 2024-03-28T15:53:06.366 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.358+0000 7f4365dc3640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 0x7f434c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.359+0000 7f4365dc3640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 0x7f434c0a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f435c0058c0 tx=0x7f435c005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:06.367 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.366+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f43500bcd30 con 0x7f43601017d0 2024-03-28T15:53:06.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.453 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: Standby manager daemon y restarted 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: Standby manager daemon y started 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: Standby manager daemon z restarted 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: Standby manager daemon z started 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:06.454 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.455 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:06.455 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.455 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:06 smithi023 ceph-mon[32448]: mgrmap e21: x(active, since 1.01861s), standbys: y, z 2024-03-28T15:53:06.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.473 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: Standby manager daemon y restarted 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: Standby manager daemon y started 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: Standby manager daemon z restarted 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: Standby manager daemon z started 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:06.474 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.475 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:06.475 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.475 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:06 smithi117 ceph-mon[36002]: mgrmap e21: x(active, since 1.01861s), standbys: y, z 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: Standby manager daemon y restarted 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: Standby manager daemon y started 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.117:0/2742562900' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.564 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: Standby manager daemon z restarted 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: Standby manager daemon z started 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: from='mgr.? 172.21.15.120:0/4155680561' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-28T15:53:06.565 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:06 smithi120 ceph-mon[36826]: mgrmap e21: x(active, since 1.01861s), standbys: y, z 2024-03-28T15:53:06.576 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.576+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f4360061c40 con 0x7f434c0a26b0 2024-03-28T15:53:06.636 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.636+0000 7f4356ffd640 1 -- 172.21.15.23:0/2679100804 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4360061c40 con 0x7f434c0a26b0 2024-03-28T15:53:06.637 INFO:teuthology.orchestra.run.smithi023.stdout:Retention added to path / 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 msgr2=0x7f434c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 0x7f434c0a4b70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f435c0058c0 tx=0x7f435c005f90 comp rx=0 tx=0).stop 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 msgr2=0x7f436019c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f436019c660 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f4350009230 tx=0x7f435002c650 comp rx=0 tx=0).stop 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 shutdown_connections 2024-03-28T15:53:06.639 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f434c0a26b0 0x7f434c0a4b70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f4360102190 0x7f436019ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43601017d0 0x7f436019c660 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 --2- 172.21.15.23:0/2679100804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43601005e0 0x7f436006a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 >> 172.21.15.23:0/2679100804 conn(0x7f43600fbb50 msgr2=0x7f4360104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 shutdown_connections 2024-03-28T15:53:06.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.638+0000 7f4368027640 1 -- 172.21.15.23:0/2679100804 wait complete. 2024-03-28T15:53:06.649 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T15:53:06.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 -- 172.21.15.23:0/518791892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 msgr2=0x7effdc106ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 --2- 172.21.15.23:0/518791892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc106ec0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7effc4009900 tx=0x7effc402bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 -- 172.21.15.23:0/518791892 shutdown_connections 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 --2- 172.21.15.23:0/518791892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 0x7effdc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 --2- 172.21.15.23:0/518791892 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc106ec0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 --2- 172.21.15.23:0/518791892 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 0x7effdc104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 -- 172.21.15.23:0/518791892 >> 172.21.15.23:0/518791892 conn(0x7effdc0fbb30 msgr2=0x7effdc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 -- 172.21.15.23:0/518791892 shutdown_connections 2024-03-28T15:53:06.754 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.753+0000 7effe0f3b640 1 -- 172.21.15.23:0/518791892 wait complete. 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 Processor -- start 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 -- start start 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 0x7effdc197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 0x7effdc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d380 con 0x7effdc104ad0 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d200 con 0x7effdc107400 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effe0f3b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc10d500 con 0x7effdc1021a0 2024-03-28T15:53:06.757 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effd3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56886/0 (socket says 172.21.15.23:56886) 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effd3fff640 1 -- 172.21.15.23:0/275643215 learned_addr learned my addr 172.21.15.23:0/275643215 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effda59c640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 0x7effdc197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.754+0000 7effdad9d640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 0x7effdc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 -- 172.21.15.23:0/275643215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 msgr2=0x7effdc197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 0x7effdc197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 -- 172.21.15.23:0/275643215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 msgr2=0x7effdc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 0x7effdc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 -- 172.21.15.23:0/275643215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc19f5b0 con 0x7effdc104ad0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd3fff640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7effc4000c00 tx=0x7effc402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4043020 con 0x7effdc104ad0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effc40054e0 con 0x7effdc104ad0 2024-03-28T15:53:06.758 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc19f840 con 0x7effdc104ad0 2024-03-28T15:53:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4003c40 con 0x7effdc104ad0 2024-03-28T15:53:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.755+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc19fd90 con 0x7effdc104ad0 2024-03-28T15:53:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.756+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc0fd620 con 0x7effdc104ad0 2024-03-28T15:53:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.757+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7effc4046020 con 0x7effdc104ad0 2024-03-28T15:53:06.759 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.758+0000 7effd37fe640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 0x7effc80a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.758+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7effc40f0b10 con 0x7effdc104ad0 2024-03-28T15:53:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.758+0000 7effda59c640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 0x7effc80a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:06.760 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.758+0000 7effda59c640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 0x7effc80a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7effcc009bd0 tx=0x7effcc005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:06.762 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.762+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7effc404d020 con 0x7effdc104ad0 2024-03-28T15:53:06.979 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.979+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7effdc061ae0 con 0x7effc80a26b0 2024-03-28T15:53:06.988 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.988+0000 7effd37fe640 1 -- 172.21.15.23:0/275643215 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7effdc061ae0 con 0x7effc80a26b0 2024-03-28T15:53:06.988 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T15:53:06", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.990+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 msgr2=0x7effc80a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.990+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 0x7effc80a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7effcc009bd0 tx=0x7effcc005f90 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.990+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 msgr2=0x7effdc197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.990+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7effc4000c00 tx=0x7effc402c650 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 shutdown_connections 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7effc80a26b0 0x7effc80a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effdc107400 0x7effdc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7effdc104ad0 0x7effdc197ed0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 --2- 172.21.15.23:0/275643215 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7effdc1021a0 0x7effdc197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 >> 172.21.15.23:0/275643215 conn(0x7effdc0fbb30 msgr2=0x7effdc102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:06.991 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 shutdown_connections 2024-03-28T15:53:06.992 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:06.991+0000 7effe0f3b640 1 -- 172.21.15.23:0/275643215 wait complete. 2024-03-28T15:53:07.003 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T15:53:07.127 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.126+0000 7f899c577640 1 -- 172.21.15.23:0/3172275172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8994100600 msgr2=0x7f8994100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 --2- 172.21.15.23:0/3172275172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8994100600 0x7f8994100a20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8990009900 tx=0x7f899002bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 -- 172.21.15.23:0/3172275172 shutdown_connections 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 --2- 172.21.15.23:0/3172275172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89941021b0 0x7f8994108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 --2- 172.21.15.23:0/3172275172 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941017f0 0x7f8994101c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 --2- 172.21.15.23:0/3172275172 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8994100600 0x7f8994100a20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 -- 172.21.15.23:0/3172275172 >> 172.21.15.23:0/3172275172 conn(0x7f89940fbb70 msgr2=0x7f89940fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 -- 172.21.15.23:0/3172275172 shutdown_connections 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 -- 172.21.15.23:0/3172275172 wait complete. 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.127+0000 7f899c577640 1 Processor -- start 2024-03-28T15:53:07.128 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 -- start start 2024-03-28T15:53:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8994100600 0x7f899406a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:07.130 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 0x7f899419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899410c270 con 0x7f89941017f0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:56912/0 (socket says 172.21.15.23:56912) 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899410c0f0 con 0x7f8994100600 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899a313640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8994100600 0x7f899406a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 -- 172.21.15.23:0/1877569063 learned_addr learned my addr 172.21.15.23:0/1877569063 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f899410c3f0 con 0x7f89941021b0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899ab14640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 0x7f899419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 -- 172.21.15.23:0/1877569063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 msgr2=0x7f899419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 0x7f899419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 -- 172.21.15.23:0/1877569063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8994100600 msgr2=0x7f899406a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8994100600 0x7f899406a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f8999b12640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899419f630 con 0x7f89941017f0 2024-03-28T15:53:07.131 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.128+0000 7f899ab14640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 0x7f899419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f8999b12640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f898400e9f0 tx=0x7f898400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898400cd80 con 0x7f89941017f0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8984005500 con 0x7f89941017f0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f899419f8c0 con 0x7f89941017f0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8984010690 con 0x7f89941017f0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.129+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899419fdc0 con 0x7f89941017f0 2024-03-28T15:53:07.132 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.130+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8994107920 con 0x7f89941017f0 2024-03-28T15:53:07.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.131+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 21) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f89840046c0 con 0x7f89941017f0 2024-03-28T15:53:07.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.132+0000 7f898b7fe640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 0x7f89800a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:07.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.132+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f89840cef40 con 0x7f89941017f0 2024-03-28T15:53:07.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.132+0000 7f899a313640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 0x7f89800a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:07.133 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.132+0000 7f899a313640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 0x7f89800a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8990000c00 tx=0x7f8990005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:07.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.137+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f898409bde0 con 0x7f89941017f0 2024-03-28T15:53:07.201 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.201+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mgrmap(e 22) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f8984097d50 con 0x7f89941017f0 2024-03-28T15:53:07.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: [28/Mar/2024:15:53:05] ENGINE Bus STARTING 2024-03-28T15:53:07.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: [28/Mar/2024:15:53:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T15:53:07.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: [28/Mar/2024:15:53:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T15:53:07.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: [28/Mar/2024:15:53:05] ENGINE Bus STARTED 2024-03-28T15:53:07.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:07.421 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:07 smithi023 ceph-mon[32448]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:07.428 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.428+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8994061ae0 con 0x7f89800a26b0 2024-03-28T15:53:07.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.435+0000 7f898b7fe640 1 -- 172.21.15.23:0/1877569063 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f8994061ae0 con 0x7f89800a26b0 2024-03-28T15:53:07.435 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-28T15:53:07.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 msgr2=0x7f89800a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:07.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 0x7f89800a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8990000c00 tx=0x7f8990005fd0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 msgr2=0x7f899419c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f898400e9f0 tx=0x7f898400edf0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 shutdown_connections 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f89800a26b0 0x7f89800a4b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f89941021b0 0x7f899419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f89941017f0 0x7f899419c700 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 --2- 172.21.15.23:0/1877569063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8994100600 0x7f899406a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 >> 172.21.15.23:0/1877569063 conn(0x7f89940fbb70 msgr2=0x7f8994104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 shutdown_connections 2024-03-28T15:53:07.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T15:53:07.437+0000 7f899c577640 1 -- 172.21.15.23:0/1877569063 wait complete. 2024-03-28T15:53:07.449 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-28T15:53:07.479 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-28T15:53:07.490 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-28T15:53:07.505 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:53:07.505 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-28T15:53:07.523 INFO:teuthology.run_tasks:Running task workunit... 2024-03-28T15:53:07.536 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:53:07.536 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-28T15:53:07.536 INFO:tasks.workunit:timeout=3h 2024-03-28T15:53:07.536 INFO:tasks.workunit:cleanup=True 2024-03-28T15:53:07.537 DEBUG:teuthology.orchestra.run.smithi023:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-28T15:53:07.551 INFO:teuthology.orchestra.run.smithi023.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-28T15:53:07.551 INFO:teuthology.orchestra.run.smithi023.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-28T15:53:07.551 INFO:teuthology.orchestra.run.smithi023.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:Access: 2024-03-28 15:51:00.314073827 +0000 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:Modify: 2024-03-28 15:51:00.314073827 +0000 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout:Change: 2024-03-28 15:52:55.141451023 +0000 2024-03-28T15:53:07.552 INFO:teuthology.orchestra.run.smithi023.stdout: Birth: 2024-03-28 15:51:00.314073827 +0000 2024-03-28T15:53:07.552 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-28T15:53:07.552 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: [28/Mar/2024:15:53:05] ENGINE Bus STARTING 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: [28/Mar/2024:15:53:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: [28/Mar/2024:15:53:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: [28/Mar/2024:15:53:05] ENGINE Bus STARTED 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:07.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:07 smithi117 ceph-mon[36002]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: [28/Mar/2024:15:53:05] ENGINE Bus STARTING 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: [28/Mar/2024:15:53:05] ENGINE Serving on http://172.21.15.23:8765 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: [28/Mar/2024:15:53:05] ENGINE Serving on https://172.21.15.23:7150 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: [28/Mar/2024:15:53:05] ENGINE Bus STARTED 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: from='client.15564 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:07.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:07 smithi120 ceph-mon[36826]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:07.625 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-28T15:53:07.680 INFO:tasks.workunit.client.0.smithi023.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-28T15:53:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:08 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:08 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: mgrmap e22: x(active, since 2s), standbys: y, z 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T15:53:08.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:08.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:08 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.925 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:09.926 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:09.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:09.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:09.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:09.927 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:09 smithi023 ceph-mon[32448]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.073 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.075 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:09 smithi117 ceph-mon[36002]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T15:53:10.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T15:53:10.092 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi023:/etc/ceph/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi117:/etc/ceph/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi120:/etc/ceph/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi023:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi120:/etc/ceph/ceph.client.admin.keyring 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.093 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:10.094 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:09 smithi120 ceph-mon[36826]: Updating smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:11.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:10 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-28T15:53:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:11.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:10 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: mgrmap e23: x(active, since 4s), standbys: y, z 2024-03-28T15:53:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:53:11.171 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:10 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:53:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:11 smithi117 ceph-mon[36002]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:11 smithi120 ceph-mon[36826]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:12.170 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:11 smithi023 ceph-mon[32448]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T15:53:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:14 smithi117 ceph-mon[36002]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T15:53:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:14 smithi120 ceph-mon[36826]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T15:53:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:14 smithi023 ceph-mon[32448]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-03-28T15:53:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:16 smithi117 ceph-mon[36002]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T15:53:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:16 smithi120 ceph-mon[36826]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T15:53:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:16 smithi023 ceph-mon[32448]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-03-28T15:53:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:18 smithi117 ceph-mon[36002]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T15:53:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:18 smithi120 ceph-mon[36826]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T15:53:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:18 smithi023 ceph-mon[32448]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-03-28T15:53:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:20 smithi117 ceph-mon[36002]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:20 smithi120 ceph-mon[36826]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.655+0000 7fddbf35b640 1 -- 172.21.15.23:0/114390661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 msgr2=0x7fddb8107a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.655+0000 7fddbf35b640 1 --2- 172.21.15.23:0/114390661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb8107a40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009930 tx=0x7fddb402bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 -- 172.21.15.23:0/114390661 shutdown_connections 2024-03-28T15:53:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 --2- 172.21.15.23:0/114390661 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 0x7fddb810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.656 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 --2- 172.21.15.23:0/114390661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb80fffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 --2- 172.21.15.23:0/114390661 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb8107a40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 -- 172.21.15.23:0/114390661 >> 172.21.15.23:0/114390661 conn(0x7fddb8076ac0 msgr2=0x7fddb80fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 -- 172.21.15.23:0/114390661 shutdown_connections 2024-03-28T15:53:20.657 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 -- 172.21.15.23:0/114390661 wait complete. 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 Processor -- start 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.656+0000 7fddbf35b640 1 -- start start 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 0x7fddb8198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8111a70 con 0x7fddb8107660 2024-03-28T15:53:20.659 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81118f0 con 0x7fddb80ffb40 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbf35b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb8111bf0 con 0x7fddb8100500 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50140/0 (socket says 172.21.15.23:50140) 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd0f7640 1 -- 172.21.15.23:0/1465729134 learned_addr learned my addr 172.21.15.23:0/1465729134 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46538/0 (socket says 172.21.15.23:46538) 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 0x7fddb8198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 -- 172.21.15.23:0/1465729134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 msgr2=0x7fddb8198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 0x7fddb8198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 -- 172.21.15.23:0/1465729134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 msgr2=0x7fddb8197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb8197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 -- 172.21.15.23:0/1465729134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb819cf00 con 0x7fddb8107660 2024-03-28T15:53:20.660 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009fd0 tx=0x7fddb000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.657+0000 7fddae7fc640 1 -- 172.21.15.23:0/1465729134 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb000ce20 con 0x7fddb8107660 2024-03-28T15:53:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.658+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb819d1f0 con 0x7fddb8107660 2024-03-28T15:53:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.658+0000 7fddae7fc640 1 -- 172.21.15.23:0/1465729134 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb0005020 con 0x7fddb8107660 2024-03-28T15:53:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.658+0000 7fddae7fc640 1 -- 172.21.15.23:0/1465729134 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0010720 con 0x7fddb8107660 2024-03-28T15:53:20.661 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.658+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb81a4a80 con 0x7fddb8107660 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.660+0000 7fddc2029740 1 Processor -- start 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddae7fc640 1 -- 172.21.15.23:0/1465729134 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fddb0005540 con 0x7fddb8107660 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 -- start start 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7e7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7f0e00 0x55771b7f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b74bb10 con 0x55771b7e7710 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b6f4900 con 0x55771b7f0e00 2024-03-28T15:53:20.669 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddc2029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b747bb0 con 0x55771b7e8030 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7e7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:50836/0 (socket says 172.21.15.23:50836) 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7e7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46546/0 (socket says 172.21.15.23:46546) 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbd0f7640 1 -- 172.21.15.23:0/2111447755 learned_addr learned my addr 172.21.15.23:0/2111447755 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7f0e00 0x55771b7f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddae7fc640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 0x7fdda40a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbd0f7640 1 -- 172.21.15.23:0/2111447755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7f0e00 msgr2=0x55771b7f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.661+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7f0e00 0x55771b7f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd0f7640 1 -- 172.21.15.23:0/2111447755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 msgr2=0x55771b7e7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7e7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 0x7fdda40a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd0f7640 1 -- 172.21.15.23:0/2111447755 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55771b460210 con 0x55771b7e8030 2024-03-28T15:53:20.670 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddae7fc640 1 -- 172.21.15.23:0/1465729134 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fddb00cf990 con 0x7fddb8107660 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009230 tx=0x7fddb4005e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a44ef894680a4c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fddbd8f8640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 0x7fdda40a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddb00cccb0 tx=0x7fddb00865e0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fdda37fe640 1 -- 172.21.15.23:0/2111447755 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb4042070 con 0x55771b7e8030 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fdda37fe640 1 -- 172.21.15.23:0/2111447755 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb4004030 con 0x55771b7e8030 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.662+0000 7fdda37fe640 1 -- 172.21.15.23:0/2111447755 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb4004c20 con 0x55771b7e8030 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- 172.21.15.23:0/2111447755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 msgr2=0x55771b7f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009230 tx=0x7fddb4005e00 comp rx=0 tx=0).stop 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- 172.21.15.23:0/2111447755 shutdown_connections 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7f0e00 0x55771b7f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e8030 0x55771b7f08c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 --2- 172.21.15.23:0/2111447755 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7e7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- 172.21.15.23:0/2111447755 >> 172.21.15.23:0/2111447755 conn(0x55771b7e2030 msgr2=0x55771b7e2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- 172.21.15.23:0/2111447755 shutdown_connections 2024-03-28T15:53:20.671 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- 172.21.15.23:0/2111447755 wait complete. 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 -- start start 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.663+0000 7fddc2029740 1 Processor -- start 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7ee330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7ee870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b7f39d0 con 0x55771b7e7710 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b707d50 con 0x55771b7e8030 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55771b7f3b50 con 0x55771b7f0e00 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 10 client.0 _ll_register_callbacks cb 0x55771b707720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46560/0 (socket says 172.21.15.23:46560) 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 -- 172.21.15.23:0/3526395901 learned_addr learned my addr 172.21.15.23:0/3526395901 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7ee330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.672 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7ee870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 -- 172.21.15.23:0/3526395901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 msgr2=0x55771b7ee870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7ee870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 -- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 msgr2=0x55771b7ee330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7ee330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 -- 172.21.15.23:0/3526395901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55771b52c620 con 0x55771b7e7710 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdda8013430 tx=0x7fdda8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2adf50e8ab2fc58 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fdda0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8004050 con 0x55771b7e7710 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fdda8009040 con 0x55771b7e7710 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.664+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddb4004030 con 0x55771b7e7710 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdda8029050 con 0x55771b7e7710 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 10 client.15636 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:20.673 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55771b7ffc20 con 0x55771b7e7710 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb4004030 con 0x55771b7e7710 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fdda80223e0 con 0x55771b7e7710 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 10 client.15636 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 10 client.15636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 20 client.15636 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b8010e0 0x55771b8034d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 4 client.15636 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55771b759050 con 0x55771b8010e0 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.665+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b8010e0 0x55771b8034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.666+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b8010e0 0x55771b8034d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac9b0fee3dfc2309 server_cookie=37a5bb4364bc6864 in_seq=0 out_seq=0 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.666+0000 7fdda0ff9640 10 client.15636 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fdda0ff9640 1 -- 172.21.15.23:0/3526395901 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55771b759050 con 0x55771b8010e0 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fdda0ff9640 10 client.15636 handle_command_reply: tid=0 2024-03-28T15:53:20.674 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 client.15636 shutdown 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 msgr2=0x55771b7eddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdda8013430 tx=0x7fdda8005f90 comp rx=0 tx=0).stop 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 shutdown_connections 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b8010e0 0x55771b8034d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7ee870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7ee330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fdda0ff9640 0 client.15636 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 --2- 172.21.15.23:0/3526395901 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7eddf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 >> 172.21.15.23:0/3526395901 conn(0x55771b7e2030 msgr2=0x55771b7e3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 shutdown_connections 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 1 -- 172.21.15.23:0/3526395901 wait complete. 2024-03-28T15:53:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.667+0000 7fddc2029740 20 client.15636 trim_cache size 0 max 16384 2024-03-28T15:53:20.676 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:20 smithi023 ceph-mon[32448]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 Processor -- start 2024-03-28T15:53:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 -- start start 2024-03-28T15:53:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e7710 0x55771b7fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81aea00 con 0x55771b7f0e00 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81b0380 con 0x55771b7e8030 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddc2029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81af060 con 0x55771b7e7710 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46574/0 (socket says 172.21.15.23:46574) 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:50166/0 (socket says 172.21.15.23:50166) 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbc8f6640 1 -- 172.21.15.23:0/3496682482 learned_addr learned my addr 172.21.15.23:0/3496682482 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e7710 0x55771b7fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 -- 172.21.15.23:0/3496682482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e7710 msgr2=0x55771b7fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e7710 0x55771b7fc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 -- 172.21.15.23:0/3496682482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 msgr2=0x55771b7ef510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbd0f7640 1 -- 172.21.15.23:0/3496682482 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55771b7e2dc0 con 0x55771b7e8030 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.681+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009430 tx=0x7fddb4004860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e22fb2d0ca4e1bea server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fdda2ffd640 1 -- 172.21.15.23:0/3496682482 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb404b070 con 0x55771b7e8030 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fdda2ffd640 1 -- 172.21.15.23:0/3496682482 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb4009520 con 0x55771b7e8030 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fdda2ffd640 1 -- 172.21.15.23:0/3496682482 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb40504b0 con 0x55771b7e8030 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fddc2029740 1 -- 172.21.15.23:0/3496682482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 msgr2=0x55771b7fc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.682+0000 7fddc2029740 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fddb4009430 tx=0x7fddb4004860 comp rx=0 tx=0).stop 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- 172.21.15.23:0/3496682482 shutdown_connections 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7f0e00 0x55771b7ef510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7fc7a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- 172.21.15.23:0/3496682482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7e7710 0x55771b7fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- 172.21.15.23:0/3496682482 >> 172.21.15.23:0/3496682482 conn(0x55771b7e2030 msgr2=0x55771b7e8640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- 172.21.15.23:0/3496682482 shutdown_connections 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- 172.21.15.23:0/3496682482 wait complete. 2024-03-28T15:53:20.688 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- start start 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 Processor -- start 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7eac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7eb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81afd20 con 0x55771b7e7710 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81af6c0 con 0x55771b7e8030 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb81acbc0 con 0x55771b7f0e00 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddc2029740 10 client.0 _ll_register_callbacks cb 0x7fdda80137c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddbd0f7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7eac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.683+0000 7fddbc8f6640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:46586/0 (socket says 172.21.15.23:46586) 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 -- 172.21.15.23:0/1535439554 learned_addr learned my addr 172.21.15.23:0/1535439554 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:20.689 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7eb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 -- 172.21.15.23:0/1535439554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 msgr2=0x55771b7eb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7eb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 -- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 msgr2=0x55771b7eac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7eac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 -- 172.21.15.23:0/1535439554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55771b85bb80 con 0x55771b7e7710 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddbc8f6640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdda8022880 tx=0x7fdda8009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6814d50cbf44958 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fdda0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8022cc0 con 0x55771b7e7710 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fdda8004030 con 0x55771b7e7710 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddb4009520 con 0x55771b7e7710 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdda803d050 con 0x55771b7e7710 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddc2029740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:20.690 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.684+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55771b85ce70 con 0x55771b7e7710 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddb4009520 con 0x55771b7e7710 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fdda80046c0 con 0x55771b7e7710 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 20 client.15642 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b85d480 0x55771b85f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 4 client.15642 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55771b7533a0 con 0x55771b85d480 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b85d480 0x55771b85f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fdda0ff9640 10 client.15642 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:20.691 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.685+0000 7fddbd0f7640 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b85d480 0x55771b85f870 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f97bd76a7d2c696 server_cookie=3d5169996292d511 in_seq=0 out_seq=0 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fdda0ff9640 1 -- 172.21.15.23:0/1535439554 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55771b7533a0 con 0x55771b85d480 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fdda0ff9640 10 client.15642 handle_command_reply: tid=0 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 client.15642 shutdown 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 msgr2=0x55771b7ea740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdda8022880 tx=0x7fdda8009b50 comp rx=0 tx=0).stop 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 shutdown_connections 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55771b85d480 0x55771b85f870 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55771b7f0e00 0x55771b7eb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55771b7e8030 0x55771b7eac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fdda0ff9640 0 client.15642 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 --2- 172.21.15.23:0/1535439554 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55771b7e7710 0x55771b7ea740 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 >> 172.21.15.23:0/1535439554 conn(0x55771b7e2030 msgr2=0x55771b7eddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 shutdown_connections 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 1 -- 172.21.15.23:0/1535439554 wait complete. 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.697+0000 7fddc2029740 20 client.15642 trim_cache size 0 max 16384 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:53:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:53:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 msgr2=0x7fdda40a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 0x7fdda40a4f70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fddb00cccb0 tx=0x7fddb00865e0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 msgr2=0x7fddb819c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fddb0009fd0 tx=0x7fddb000ef10 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 shutdown_connections 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda40a2ab0 0x7fdda40a4f70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb8107660 0x7fddb819c7d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb8100500 0x7fddb8198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 --2- 172.21.15.23:0/1465729134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb80ffb40 0x7fddb8197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 >> 172.21.15.23:0/1465729134 conn(0x7fddb8076ac0 msgr2=0x7fddb8104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 shutdown_connections 2024-03-28T15:53:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:20.698+0000 7fddbf35b640 1 -- 172.21.15.23:0/1465729134 wait complete. 2024-03-28T15:53:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:22 smithi117 ceph-mon[36002]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:22 smithi120 ceph-mon[36826]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:22 smithi023 ceph-mon[32448]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-03-28T15:53:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:24 smithi117 ceph-mon[36002]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-28T15:53:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:24 smithi120 ceph-mon[36826]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-28T15:53:24.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:24 smithi023 ceph-mon[32448]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 59 op/s 2024-03-28T15:53:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:26 smithi117 ceph-mon[36002]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:26 smithi120 ceph-mon[36826]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:26 smithi023 ceph-mon[32448]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:28 smithi117 ceph-mon[36002]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:28 smithi120 ceph-mon[36826]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:28 smithi023 ceph-mon[32448]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:30.488 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:30 smithi023 ceph-mon[32448]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:30 smithi117 ceph-mon[36002]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:30 smithi120 ceph-mon[36826]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-28T15:53:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:32 smithi117 ceph-mon[36002]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T15:53:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:32 smithi120 ceph-mon[36826]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T15:53:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:32 smithi023 ceph-mon[32448]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.7 KiB/s wr, 15 op/s 2024-03-28T15:53:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 -- 172.21.15.23:0/2144754189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e4068d20 msgr2=0x7f42e4069180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 --2- 172.21.15.23:0/2144754189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e4068d20 0x7f42e4069180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009930 tx=0x7f42d402bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 -- 172.21.15.23:0/2144754189 shutdown_connections 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 --2- 172.21.15.23:0/2144754189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e40696c0 0x7f42e410e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 --2- 172.21.15.23:0/2144754189 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e4068d20 0x7f42e4069180 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 --2- 172.21.15.23:0/2144754189 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 0x7f42e4107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 -- 172.21.15.23:0/2144754189 >> 172.21.15.23:0/2144754189 conn(0x7f42e40773f0 msgr2=0x7f42e4077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 -- 172.21.15.23:0/2144754189 shutdown_connections 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.786+0000 7f42eb426640 1 -- 172.21.15.23:0/2144754189 wait complete. 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 Processor -- start 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 -- start start 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e4068d20 0x7f42e4197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 0x7f42e419c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4111a60 con 0x7f42e40696c0 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41118e0 con 0x7f42e4068d20 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42eb426640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e4111be0 con 0x7f42e4107770 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37452/0 (socket says 172.21.15.23:37452) 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 -- 172.21.15.23:0/3595773161 learned_addr learned my addr 172.21.15.23:0/3595773161 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e91c2640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e4068d20 0x7f42e4197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e99c3640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 0x7f42e419c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 -- 172.21.15.23:0/3595773161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 msgr2=0x7f42e419c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 0x7f42e419c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 -- 172.21.15.23:0/3595773161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e4068d20 msgr2=0x7f42e4197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e4068d20 0x7f42e4197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.787+0000 7f42e89c1640 1 -- 172.21.15.23:0/3595773161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e419ce90 con 0x7f42e40696c0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009230 tx=0x7f42d40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42da7fc640 1 -- 172.21.15.23:0/3595773161 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4041070 con 0x7f42e40696c0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e419d120 con 0x7f42e40696c0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42da7fc640 1 -- 172.21.15.23:0/3595773161 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42d402ccc0 con 0x7f42e40696c0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42da7fc640 1 -- 172.21.15.23:0/3595773161 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d4003de0 con 0x7f42e40696c0 2024-03-28T15:53:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.788+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e41a4a10 con 0x7f42e40696c0 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.790+0000 7f42da7fc640 1 -- 172.21.15.23:0/3595773161 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f42d4044070 con 0x7f42e40696c0 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.790+0000 7f42ee00c740 1 Processor -- start 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.790+0000 7f42ee00c740 1 -- start start 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x5647792d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56477922bb10 con 0x5647792c8030 2024-03-28T15:53:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647791d4900 con 0x5647792c7710 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42ee00c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x564779227bb0 con 0x5647792d0e00 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37468/0 (socket says 172.21.15.23:37468) 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e91c2640 1 -- 172.21.15.23:0/364735696 learned_addr learned my addr 172.21.15.23:0/364735696 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e89c1640 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x5647792d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42da7fc640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 0x7f42d00a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 -- 172.21.15.23:0/364735696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 msgr2=0x5647792d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x5647792d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 -- 172.21.15.23:0/364735696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 msgr2=0x5647792d08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 -- 172.21.15.23:0/364735696 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564778f40210 con 0x5647792c7710 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42da7fc640 1 -- 172.21.15.23:0/3595773161 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f42d40f0ff0 con 0x7f42e40696c0 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 0x7f42d00a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.791+0000 7f42e99c3640 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792c7af0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f42dc003b10 tx=0x7f42dc004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78ca99829721755 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42cf7fe640 1 -- 172.21.15.23:0/364735696 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc024070 con 0x5647792c7710 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42cf7fe640 1 -- 172.21.15.23:0/364735696 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42dc004d20 con 0x5647792c7710 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42e99c3640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 0x7f42d00a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42dc021000 tx=0x7f42dc01e000 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42cf7fe640 1 -- 172.21.15.23:0/364735696 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc005450 con 0x5647792c7710 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 -- 172.21.15.23:0/364735696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 msgr2=0x5647792c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792c7af0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f42dc003b10 tx=0x7f42dc004040 comp rx=0 tx=0).stop 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 -- 172.21.15.23:0/364735696 shutdown_connections 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x5647792d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.800 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 --2- 172.21.15.23:0/364735696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792c7af0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 -- 172.21.15.23:0/364735696 >> 172.21.15.23:0/364735696 conn(0x5647792c2030 msgr2=0x5647792c2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 -- 172.21.15.23:0/364735696 shutdown_connections 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.792+0000 7f42ee00c740 1 -- 172.21.15.23:0/364735696 wait complete. 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 -- start start 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 Processor -- start 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c8030 0x5647792ce3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792d0e00 0x5647792ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.801 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647792d3a30 con 0x5647792c7710 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647791e7d50 con 0x5647792d0e00 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.793+0000 7f42ee00c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647792d3bb0 con 0x5647792c8030 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ee00c740 10 client.0 _ll_register_callbacks cb 0x5647791e7720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37484/0 (socket says 172.21.15.23:37484) 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 -- 172.21.15.23:0/3250506813 learned_addr learned my addr 172.21.15.23:0/3250506813 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c8030 0x5647792ce3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e99c3640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792d0e00 0x5647792ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 -- 172.21.15.23:0/3250506813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c8030 msgr2=0x5647792ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c8030 0x5647792ce3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 -- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792d0e00 msgr2=0x5647792ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792d0e00 0x5647792ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.802 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 -- 172.21.15.23:0/3250506813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56477900c620 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42e91c2640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f42e0012720 tx=0x7f42e0012b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408cfe554ee84c61 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0004050 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f42e00041f0 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42dc004d20 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42e0025050 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ee00c740 10 client.15654 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.794+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5647792dfd50 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42dc004d20 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f42e0003a30 con 0x5647792c7710 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 10 client.15654 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 10 client.15654 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 20 client.15654 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:33.803 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5647792e1210 0x5647792e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 4 client.15654 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564779239050 con 0x5647792e1210 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5647792e1210 0x5647792e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42ccff9640 10 client.15654 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.795+0000 7f42e89c1640 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5647792e1210 0x5647792e3600 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2f9ad04ffb95dd0 server_cookie=88bcde58ac61e6e2 in_seq=0 out_seq=0 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ccff9640 1 -- 172.21.15.23:0/3250506813 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564779239050 con 0x5647792e1210 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ccff9640 10 client.15654 handle_command_reply: tid=0 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 client.15654 shutdown 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 msgr2=0x5647792cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f42e0012720 tx=0x7f42e0012b20 comp rx=0 tx=0).stop 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 shutdown_connections 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5647792e1210 0x5647792e3600 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792d0e00 0x5647792ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c8030 0x5647792ce3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.804 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ccff9640 0 client.15654 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 --2- 172.21.15.23:0/3250506813 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c7710 0x5647792cdea0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 >> 172.21.15.23:0/3250506813 conn(0x5647792c2030 msgr2=0x5647792c3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 shutdown_connections 2024-03-28T15:53:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 1 -- 172.21.15.23:0/3250506813 wait complete. 2024-03-28T15:53:33.805 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.796+0000 7f42ee00c740 20 client.15654 trim_cache size 0 max 16384 2024-03-28T15:53:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.810+0000 7f42ee00c740 1 Processor -- start 2024-03-28T15:53:33.815 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.810+0000 7f42ee00c740 1 -- start start 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c7710 0x5647792d05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c8030 0x5647792dc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41ae900 con 0x5647792d0e00 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41b0280 con 0x5647792c8030 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ee00c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41aef60 con 0x5647792c7710 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37498/0 (socket says 172.21.15.23:37498) 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 -- 172.21.15.23:0/2481505637 learned_addr learned my addr 172.21.15.23:0/2481505637 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e89c1640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c8030 0x5647792dc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c7710 0x5647792d05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 -- 172.21.15.23:0/2481505637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c7710 msgr2=0x5647792d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.816 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c7710 0x5647792d05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 -- 172.21.15.23:0/2481505637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c8030 msgr2=0x5647792dc2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c8030 0x5647792dc2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 -- 172.21.15.23:0/2481505637 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647792c7e60 con 0x5647792d0e00 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42e99c3640 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f42dc010fd0 tx=0x7f42dc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d5bc2aed14bd55 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ceffd640 1 -- 172.21.15.23:0/2481505637 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc024030 con 0x5647792d0e00 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ceffd640 1 -- 172.21.15.23:0/2481505637 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42dc005160 con 0x5647792d0e00 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.811+0000 7f42ceffd640 1 -- 172.21.15.23:0/2481505637 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc004c30 con 0x5647792d0e00 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- 172.21.15.23:0/2481505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 msgr2=0x5647792dc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f42dc010fd0 tx=0x7f42dc005f70 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- 172.21.15.23:0/2481505637 shutdown_connections 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792d0e00 0x5647792dc800 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c8030 0x5647792dc2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 --2- 172.21.15.23:0/2481505637 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792c7710 0x5647792d05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- 172.21.15.23:0/2481505637 >> 172.21.15.23:0/2481505637 conn(0x5647792c2030 msgr2=0x5647792c2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.817 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- 172.21.15.23:0/2481505637 shutdown_connections 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- 172.21.15.23:0/2481505637 wait complete. 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 -- start start 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.812+0000 7f42ee00c740 1 Processor -- start 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792d4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x564779335ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41afc20 con 0x5647792c8030 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41af5c0 con 0x5647792c7710 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41acac0 con 0x5647792d0e00 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 10 client.0 _ll_register_callbacks cb 0x7f42e0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:37504/0 (socket says 172.21.15.23:37504) 2024-03-28T15:53:33.818 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 -- 172.21.15.23:0/1678646604 learned_addr learned my addr 172.21.15.23:0/1678646604 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e91c2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792d4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e99c3640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x564779335ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 -- 172.21.15.23:0/1678646604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 msgr2=0x564779335ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x564779335ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 -- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 msgr2=0x5647792d4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792d4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 -- 172.21.15.23:0/1678646604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564779336100 con 0x5647792c8030 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42e89c1640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f42d40bd260 tx=0x7f42d40f3500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af979593539b40 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d404b040 con 0x5647792c8030 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f42d40035f0 con 0x5647792c8030 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.813+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42dc005160 con 0x5647792c8030 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42d40bce10 con 0x5647792c8030 2024-03-28T15:53:33.819 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 10 client.15660 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56477933d4e0 con 0x5647792c8030 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42dc005160 con 0x5647792c8030 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f42d40b9940 con 0x5647792c8030 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 10 client.15660 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 10 client.15660 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 20 client.15660 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x56477933dbc0 0x56477933ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 4 client.15660 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5647792333a0 con 0x56477933dbc0 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.814+0000 7f42e91c2640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x56477933dbc0 0x56477933ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.815+0000 7f42e91c2640 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x56477933dbc0 0x56477933ffe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba56d7524db087 server_cookie=7b52212bacb8e176 in_seq=0 out_seq=0 2024-03-28T15:53:33.820 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.815+0000 7f42ccff9640 10 client.15660 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ccff9640 1 -- 172.21.15.23:0/1678646604 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5647792333a0 con 0x56477933dbc0 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ccff9640 10 client.15660 handle_command_reply: tid=0 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 client.15660 shutdown 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 msgr2=0x5647792d4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f42d40bd260 tx=0x7f42d40f3500 comp rx=0 tx=0).stop 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 shutdown_connections 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x56477933dbc0 0x56477933ffe0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5647792d0e00 0x564779335ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5647792c8030 0x5647792d4fb0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 --2- 172.21.15.23:0/1678646604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5647792c7710 0x5647792d4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 >> 172.21.15.23:0/1678646604 conn(0x5647792c2030 msgr2=0x5647792c0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.822 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ccff9640 0 client.15660 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 shutdown_connections 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.819+0000 7f42ee00c740 1 -- 172.21.15.23:0/1678646604 wait complete. 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42ee00c740 20 client.15660 trim_cache size 0 max 16384 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:53:33.823 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 msgr2=0x7f42d00a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 0x7f42d00a4f70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f42dc021000 tx=0x7f42dc01e000 comp rx=0 tx=0).stop 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 msgr2=0x7f42e41983e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009230 tx=0x7f42d40043b0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 shutdown_connections 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f42d00a2ab0 0x7f42d00a4f70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.824 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f42e4107770 0x7f42e419c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f42e40696c0 0x7f42e41983e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 --2- 172.21.15.23:0/3595773161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42e4068d20 0x7f42e4197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:33.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 >> 172.21.15.23:0/3595773161 conn(0x7f42e40773f0 msgr2=0x7f42e40fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:33.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 shutdown_connections 2024-03-28T15:53:33.825 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:33.820+0000 7f42eb426640 1 -- 172.21.15.23:0/3595773161 wait complete. 2024-03-28T15:53:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:34 smithi117 ceph-mon[36002]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:34 smithi120 ceph-mon[36826]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:34 smithi023 ceph-mon[32448]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.7 KiB/s wr, 17 op/s 2024-03-28T15:53:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:36 smithi117 ceph-mon[36002]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:53:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:36 smithi120 ceph-mon[36826]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:53:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:36 smithi023 ceph-mon[32448]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:53:38.575 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:38 smithi117 ceph-mon[36002]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-28T15:53:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:38 smithi120 ceph-mon[36826]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-28T15:53:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:38 smithi023 ceph-mon[32448]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 341 B/s wr, 13 op/s 2024-03-28T15:53:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:40 smithi117 ceph-mon[36002]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T15:53:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:40 smithi120 ceph-mon[36826]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T15:53:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:40 smithi023 ceph-mon[32448]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-28T15:53:42.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:42 smithi117 ceph-mon[36002]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T15:53:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:42 smithi120 ceph-mon[36826]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T15:53:42.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:42 smithi023 ceph-mon[32448]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-28T15:53:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:44 smithi117 ceph-mon[36002]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T15:53:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:44 smithi120 ceph-mon[36826]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T15:53:44.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:44 smithi023 ceph-mon[32448]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-03-28T15:53:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:46 smithi117 ceph-mon[36002]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:53:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:46 smithi120 ceph-mon[36826]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:53:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:46 smithi023 ceph-mon[32448]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 -- 172.21.15.23:0/3216766533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22281007f0 msgr2=0x7f2228100c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 --2- 172.21.15.23:0/3216766533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22281007f0 0x7f2228100c70 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f221c009900 tx=0x7f221c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 -- 172.21.15.23:0/3216766533 shutdown_connections 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 --2- 172.21.15.23:0/3216766533 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281011b0 0x7f222810d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 --2- 172.21.15.23:0/3216766533 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f22281007f0 0x7f2228100c70 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 --2- 172.21.15.23:0/3216766533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2228105540 0x7f2228105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 -- 172.21.15.23:0/3216766533 >> 172.21.15.23:0/3216766533 conn(0x7f22280773f0 msgr2=0x7f2228077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 -- 172.21.15.23:0/3216766533 shutdown_connections 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.908+0000 7f222cc34640 1 -- 172.21.15.23:0/3216766533 wait complete. 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 Processor -- start 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 -- start start 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 0x7f2228197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22281011b0 0x7f2228198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228111b80 con 0x7f2228105540 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228111a00 con 0x7f22281011b0 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222cc34640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2228111d00 con 0x7f22281007f0 2024-03-28T15:53:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 0x7f2228197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45814/0 (socket says 172.21.15.23:45814) 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 -- 172.21.15.23:0/2276403653 learned_addr learned my addr 172.21.15.23:0/2276403653 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f222659c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 0x7f2228197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:34814/0 (socket says 172.21.15.23:34814) 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2225d9b640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22281011b0 0x7f2228198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 -- 172.21.15.23:0/2276403653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 msgr2=0x7f2228197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 0x7f2228197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 -- 172.21.15.23:0/2276403653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22281011b0 msgr2=0x7f2228198440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22281011b0 0x7f2228198440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.909+0000 7f2226d9d640 1 -- 172.21.15.23:0/2276403653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222819cef0 con 0x7f2228105540 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f222000ea60 tx=0x7f222000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.913 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f22177fe640 1 -- 172.21.15.23:0/2276403653 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222000ce40 con 0x7f2228105540 2024-03-28T15:53:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222819d1e0 con 0x7f2228105540 2024-03-28T15:53:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f22177fe640 1 -- 172.21.15.23:0/2276403653 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2220005020 con 0x7f2228105540 2024-03-28T15:53:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f22177fe640 1 -- 172.21.15.23:0/2276403653 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220010690 con 0x7f2228105540 2024-03-28T15:53:46.914 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.910+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22281a4a70 con 0x7f2228105540 2024-03-28T15:53:46.921 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.912+0000 7f22177fe640 1 -- 172.21.15.23:0/2276403653 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 137466+0+0 (secure 0 0 0) 0x7f22200051d0 con 0x7f2228105540 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.912+0000 7f222f81a740 1 Processor -- start 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f22177fe640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 0x7f22100a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 -- start start 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f22177fe640 1 -- 172.21.15.23:0/2276403653 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f22200cfa30 con 0x7f2228105540 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2225d9b640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 0x7f22100a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa846af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa7aab10 con 0x5650aa84fe00 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa753900 con 0x5650aa847030 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222f81a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa7a6bb0 con 0x5650aa846710 2024-03-28T15:53:46.922 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2225d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45826/0 (socket says 172.21.15.23:45826) 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 -- 172.21.15.23:0/322333766 learned_addr learned my addr 172.21.15.23:0/322333766 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f222659c640 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa846af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 -- 172.21.15.23:0/322333766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 msgr2=0x5650aa846af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa846af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 -- 172.21.15.23:0/322333766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 msgr2=0x5650aa84f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 -- 172.21.15.23:0/322333766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650aa4bf210 con 0x5650aa84fe00 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2225d9b640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 0x7f22100a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f221c009230 tx=0x7f221c0041a0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.913+0000 7f2226d9d640 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2220009a30 tx=0x7f222009ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a32aa1ddfe0369 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.914+0000 7f2214ff9640 1 -- 172.21.15.23:0/322333766 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222009fca0 con 0x5650aa84fe00 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.914+0000 7f2214ff9640 1 -- 172.21.15.23:0/322333766 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22200a42f0 con 0x5650aa84fe00 2024-03-28T15:53:46.923 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.914+0000 7f2214ff9640 1 -- 172.21.15.23:0/322333766 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222009afa0 con 0x5650aa84fe00 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- 172.21.15.23:0/322333766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 msgr2=0x5650aa8521f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2220009a30 tx=0x7f222009ba50 comp rx=0 tx=0).stop 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- 172.21.15.23:0/322333766 shutdown_connections 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa8521f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- 172.21.15.23:0/322333766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa846af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- 172.21.15.23:0/322333766 >> 172.21.15.23:0/322333766 conn(0x5650aa841030 msgr2=0x5650aa841410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- 172.21.15.23:0/322333766 shutdown_connections 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- 172.21.15.23:0/322333766 wait complete. 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- start start 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 Processor -- start 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.924 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa84fe00 0x5650aa84d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa8529d0 con 0x5650aa846710 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa766d50 con 0x5650aa847030 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.915+0000 7f222f81a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650aa852b50 con 0x5650aa84fe00 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222f81a740 10 client.0 _ll_register_callbacks cb 0x5650aa766720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45836/0 (socket says 172.21.15.23:45836) 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 -- 172.21.15.23:0/1522154085 learned_addr learned my addr 172.21.15.23:0/1522154085 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f2226d9d640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa84fe00 0x5650aa84d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f2225d9b640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 -- 172.21.15.23:0/1522154085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa84fe00 msgr2=0x5650aa84d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa84fe00 0x5650aa84d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 -- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 msgr2=0x5650aa84d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 -- 172.21.15.23:0/1522154085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650aa58b620 con 0x5650aa846710 2024-03-28T15:53:46.925 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222659c640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2218012840 tx=0x7f2218012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75bf1cc0d52123d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f220dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221800f8a0 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f221800fa40 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22200a42f0 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f221800fc40 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222f81a740 10 client.15672 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5650aa85ecc0 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.916+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22200a42f0 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f221801c420 con 0x5650aa846710 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 10 client.15672 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 10 client.15672 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 20 client.15672 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa860180 0x5650aa862590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.926 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 4 client.15672 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650aa7b8050 con 0x5650aa860180 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f2225d9b640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa860180 0x5650aa862590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f220dffb640 10 client.15672 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.917+0000 7f2225d9b640 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa860180 0x5650aa862590 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15fc45a2728ea0ad server_cookie=fc556d392799d417 in_seq=0 out_seq=0 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f220dffb640 1 -- 172.21.15.23:0/1522154085 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5650aa7b8050 con 0x5650aa860180 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f220dffb640 10 client.15672 handle_command_reply: tid=0 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 client.15672 shutdown 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 msgr2=0x5650aa84ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f2218012840 tx=0x7f2218012d00 comp rx=0 tx=0).stop 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 shutdown_connections 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa860180 0x5650aa862590 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa84fe00 0x5650aa84d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa84d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.927 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 --2- 172.21.15.23:0/1522154085 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa846710 0x5650aa84ce60 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 >> 172.21.15.23:0/1522154085 conn(0x5650aa841030 msgr2=0x5650aa842d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f220dffb640 0 client.15672 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 shutdown_connections 2024-03-28T15:53:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 1 -- 172.21.15.23:0/1522154085 wait complete. 2024-03-28T15:53:46.928 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.918+0000 7f222f81a740 20 client.15672 trim_cache size 0 max 16384 2024-03-28T15:53:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 Processor -- start 2024-03-28T15:53:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 -- start start 2024-03-28T15:53:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa846710 0x5650aa84f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa847030 0x5650aa85b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281ae9d0 con 0x5650aa84fe00 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281b0350 con 0x5650aa846710 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222f81a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281af030 con 0x5650aa847030 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45852/0 (socket says 172.21.15.23:45852) 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f2226d9d640 1 -- 172.21.15.23:0/8679720 learned_addr learned my addr 172.21.15.23:0/8679720 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f2225d9b640 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa847030 0x5650aa85b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.932+0000 7f222659c640 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa846710 0x5650aa84f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 -- 172.21.15.23:0/8679720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa847030 msgr2=0x5650aa85b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa847030 0x5650aa85b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 -- 172.21.15.23:0/8679720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa846710 msgr2=0x5650aa84f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa846710 0x5650aa84f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 -- 172.21.15.23:0/8679720 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650aa8521f0 con 0x5650aa84fe00 2024-03-28T15:53:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f2226d9d640 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f22200ccdd0 tx=0x7f22200096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44870e07eb39318e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f220ffff640 1 -- 172.21.15.23:0/8679720 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200a42f0 con 0x5650aa84fe00 2024-03-28T15:53:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f220ffff640 1 -- 172.21.15.23:0/8679720 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2220099e50 con 0x5650aa84fe00 2024-03-28T15:53:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.933+0000 7f220ffff640 1 -- 172.21.15.23:0/8679720 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222009afa0 con 0x5650aa84fe00 2024-03-28T15:53:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- 172.21.15.23:0/8679720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 msgr2=0x5650aa85b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f22200ccdd0 tx=0x7f22200096c0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- 172.21.15.23:0/8679720 shutdown_connections 2024-03-28T15:53:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa85b7a0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa847030 0x5650aa85b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- 172.21.15.23:0/8679720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa846710 0x5650aa84f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- 172.21.15.23:0/8679720 >> 172.21.15.23:0/8679720 conn(0x5650aa841030 msgr2=0x5650aa840020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- 172.21.15.23:0/8679720 shutdown_connections 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- 172.21.15.23:0/8679720 wait complete. 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- start start 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 Processor -- start 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa8578d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa853c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281afcf0 con 0x5650aa84fe00 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281af690 con 0x5650aa847030 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22281acb90 con 0x5650aa846710 2024-03-28T15:53:46.939 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.934+0000 7f222f81a740 10 client.0 _ll_register_callbacks cb 0x7f2218012450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45864/0 (socket says 172.21.15.23:45864) 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2225d9b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa853c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 -- 172.21.15.23:0/2300190038 learned_addr learned my addr 172.21.15.23:0/2300190038 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f222659c640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa8578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 -- 172.21.15.23:0/2300190038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 msgr2=0x5650aa8578d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa8578d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 -- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 msgr2=0x5650aa853c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa853c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 -- 172.21.15.23:0/2300190038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650aa854750 con 0x5650aa84fe00 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f2226d9d640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2220097fb0 tx=0x7f222009b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17abd38fc5296e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f220dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222009fca0 con 0x5650aa84fe00 2024-03-28T15:53:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f22200b4070 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2220099e50 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22200a42f0 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f222f81a740 10 client.15678 fetch_fsmap learned FSMap version 29 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.935+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5650aa8c0c20 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2220099e50 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f22200a0bc0 con 0x5650aa84fe00 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 10 client.15678 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 10 client.15678 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 20 client.15678 populate_metadata read hostname 'smithi023' 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa8c0e50 0x5650aa8c3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 4 client.15678 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5650aa7b23a0 con 0x5650aa8c0e50 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222659c640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa8c0e50 0x5650aa8c3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f220dffb640 10 client.15678 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.936+0000 7f222659c640 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa8c0e50 0x5650aa8c3240 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43739c272854640f server_cookie=8317383ac6a05d in_seq=0 out_seq=0 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f220dffb640 1 -- 172.21.15.23:0/2300190038 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5650aa7b23a0 con 0x5650aa8c0e50 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f220dffb640 10 client.15678 handle_command_reply: tid=0 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 client.15678 shutdown 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 msgr2=0x5650aa854190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f2220097fb0 tx=0x7f222009b560 comp rx=0 tx=0).stop 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 shutdown_connections 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5650aa8c0e50 0x5650aa8c3240 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.943 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5650aa84fe00 0x5650aa854190 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f220dffb640 0 client.15678 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5650aa847030 0x5650aa853c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 --2- 172.21.15.23:0/2300190038 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5650aa846710 0x5650aa8578d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 >> 172.21.15.23:0/2300190038 conn(0x5650aa841030 msgr2=0x5650aa83f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.942+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 shutdown_connections 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222f81a740 1 -- 172.21.15.23:0/2300190038 wait complete. 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222f81a740 20 client.15678 trim_cache size 0 max 16384 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:53:46.944 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:53:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 msgr2=0x7f22100a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 0x7f22100a4f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f221c009230 tx=0x7f221c0041a0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 msgr2=0x7f222819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:53:46.945 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f222000ea60 tx=0x7f222000ef20 comp rx=0 tx=0).stop 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 shutdown_connections 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f22100a2ab0 0x7f22100a4f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f2228105540 0x7f222819c7c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22281011b0 0x7f2228198440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 --2- 172.21.15.23:0/2276403653 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f22281007f0 0x7f2228197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 >> 172.21.15.23:0/2276403653 conn(0x7f22280773f0 msgr2=0x7f22280fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 shutdown_connections 2024-03-28T15:53:46.946 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:53:46.943+0000 7f222cc34640 1 -- 172.21.15.23:0/2276403653 wait complete. 2024-03-28T15:53:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:48 smithi117 ceph-mon[36002]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T15:53:48.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:48 smithi120 ceph-mon[36826]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T15:53:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:48 smithi023 ceph-mon[32448]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-03-28T15:53:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:50 smithi117 ceph-mon[36002]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:53:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:50 smithi120 ceph-mon[36826]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:53:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:50 smithi023 ceph-mon[32448]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:53:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:52 smithi117 ceph-mon[36002]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-28T15:53:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:52 smithi120 ceph-mon[36826]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-28T15:53:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:52 smithi023 ceph-mon[32448]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-28T15:53:54.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:54 smithi117 ceph-mon[36002]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-28T15:53:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:54 smithi120 ceph-mon[36826]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-28T15:53:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:54 smithi023 ceph-mon[32448]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-28T15:53:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:56 smithi117 ceph-mon[36002]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:53:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:56 smithi120 ceph-mon[36826]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:53:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:56 smithi023 ceph-mon[32448]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:53:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:53:58 smithi117 ceph-mon[36002]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-28T15:53:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:53:58 smithi120 ceph-mon[36826]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-28T15:53:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:53:58 smithi023 ceph-mon[32448]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 767 B/s wr, 14 op/s 2024-03-28T15:54:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 -- 172.21.15.23:0/3654203133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c1011b0 msgr2=0x7fdd1c10d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3654203133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c10d3d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fdd14009c10 tx=0x7fdd1402bef0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 -- 172.21.15.23:0/3654203133 shutdown_connections 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3654203133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c10d3d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3654203133 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 0x7fdd1c100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3654203133 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c105540 0x7fdd1c105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 -- 172.21.15.23:0/3654203133 >> 172.21.15.23:0/3654203133 conn(0x7fdd1c0773f0 msgr2=0x7fdd1c077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 -- 172.21.15.23:0/3654203133 shutdown_connections 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.038+0000 7fdd22adc640 1 -- 172.21.15.23:0/3654203133 wait complete. 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 Processor -- start 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 -- start start 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 0x7fdd1c197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.042 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 0x7fdd1c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c111bb0 con 0x7fdd1c1011b0 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c111a30 con 0x7fdd1c105540 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd22adc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c111d30 con 0x7fdd1c1007f0 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45906/0 (socket says 172.21.15.23:45906) 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd1bfff640 1 -- 172.21.15.23:0/3407798218 learned_addr learned my addr 172.21.15.23:0/3407798218 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd21079640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 0x7fdd1c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.039+0000 7fdd20878640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 0x7fdd1c197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 -- 172.21.15.23:0/3407798218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 msgr2=0x7fdd1c197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 0x7fdd1c197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 -- 172.21.15.23:0/3407798218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 msgr2=0x7fdd1c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.043 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 0x7fdd1c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 -- 172.21.15.23:0/3407798218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1c19cef0 con 0x7fdd1c1011b0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd21079640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 0x7fdd1c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800e9f0 tx=0x7fdd0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd19ffb640 1 -- 172.21.15.23:0/3407798218 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0800ccc0 con 0x7fdd1c1011b0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd19ffb640 1 -- 172.21.15.23:0/3407798218 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd0800ce60 con 0x7fdd1c1011b0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c19d1e0 con 0x7fdd1c1011b0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd19ffb640 1 -- 172.21.15.23:0/3407798218 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08010690 con 0x7fdd1c1011b0 2024-03-28T15:54:00.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.040+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1c1a4a70 con 0x7fdd1c1011b0 2024-03-28T15:54:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.042+0000 7fdd19ffb640 1 -- 172.21.15.23:0/3407798218 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 23) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fdd080046c0 con 0x7fdd1c1011b0 2024-03-28T15:54:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.043+0000 7fdd19ffb640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 0x7fdd0c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.043+0000 7fdd20878640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 0x7fdd0c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.043+0000 7fdd19ffb640 1 -- 172.21.15.23:0/3407798218 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fdd080cf790 con 0x7fdd1c1011b0 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.043+0000 7fdd20878640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 0x7fdd0c0a4f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005fd0 tx=0x7fdd10009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 Processor -- start 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 -- start start 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe080de00 0x55ebe08101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0768b10 con 0x55ebe0804710 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0711900 con 0x55ebe0805030 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd257a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0764bb0 con 0x55ebe080de00 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45912/0 (socket says 172.21.15.23:45912) 2024-03-28T15:54:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 -- 172.21.15.23:0/2283721766 learned_addr learned my addr 172.21.15.23:0/2283721766 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd21079640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe080de00 0x55ebe08101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 -- 172.21.15.23:0/2283721766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe080de00 msgr2=0x55ebe08101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe080de00 0x55ebe08101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 -- 172.21.15.23:0/2283721766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 msgr2=0x55ebe080d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd20878640 1 -- 172.21.15.23:0/2283721766 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebe047d210 con 0x55ebe0804710 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.044+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd20878640 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdd10004310 tx=0x7fdd10014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb87b0b484ce22d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd06ffd640 1 -- 172.21.15.23:0/2283721766 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10005020 con 0x55ebe0804710 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd06ffd640 1 -- 172.21.15.23:0/2283721766 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd100047a0 con 0x55ebe0804710 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd06ffd640 1 -- 172.21.15.23:0/2283721766 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1001e820 con 0x55ebe0804710 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd257a7740 1 -- 172.21.15.23:0/2283721766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 msgr2=0x55ebe0804af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.045+0000 7fdd257a7740 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdd10004310 tx=0x7fdd10014000 comp rx=0 tx=0).stop 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 -- 172.21.15.23:0/2283721766 shutdown_connections 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe080de00 0x55ebe08101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 --2- 172.21.15.23:0/2283721766 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0804710 0x55ebe0804af0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 -- 172.21.15.23:0/2283721766 >> 172.21.15.23:0/2283721766 conn(0x55ebe07ff030 msgr2=0x55ebe07ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 -- 172.21.15.23:0/2283721766 shutdown_connections 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 -- 172.21.15.23:0/2283721766 wait complete. 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 -- start start 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 Processor -- start 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.046+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe080b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe080b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0810a30 con 0x55ebe0805030 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0724d50 con 0x55ebe080de00 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebe0810bb0 con 0x55ebe0804710 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 10 client.0 _ll_register_callbacks cb 0x55ebe0724720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe080b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe080b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:42364/0 (socket says 172.21.15.23:42364) 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd20878640 1 -- 172.21.15.23:0/3200999914 learned_addr learned my addr 172.21.15.23:0/3200999914 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 -- 172.21.15.23:0/3200999914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 msgr2=0x55ebe080ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 -- 172.21.15.23:0/3200999914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 msgr2=0x55ebe080b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe080b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 -- 172.21.15.23:0/3200999914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebe0549620 con 0x55ebe080de00 2024-03-28T15:54:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd20878640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd21079640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe080b8d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036040 tx=0x7fdd14033a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=817da10bb7bbc031 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdcfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14003970 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd100047a0 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.047+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fdd14003b10 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd14052050 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 10 client.25529 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ebe081ccb0 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd100047a0 con 0x55ebe080de00 2024-03-28T15:54:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fdd14004410 con 0x55ebe080de00 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 10 client.25529 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 10 client.25529 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 20 client.25529 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe081e170 0x55ebe0820560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 4 client.25529 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebe0776050 con 0x55ebe081e170 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.048+0000 7fdd20878640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe081e170 0x55ebe0820560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.049+0000 7fdd20878640 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe081e170 0x55ebe0820560 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4205c49109424f3c server_cookie=8c7f3266b39160c9 in_seq=0 out_seq=0 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.049+0000 7fdcfbfff640 10 client.25529 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdcfbfff640 1 -- 172.21.15.23:0/3200999914 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ebe0776050 con 0x55ebe081e170 2024-03-28T15:54:00.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdcfbfff640 10 client.25529 handle_command_reply: tid=0 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 client.25529 shutdown 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 msgr2=0x55ebe080b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe080b8d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdd14036040 tx=0x7fdd14033a30 comp rx=0 tx=0).stop 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 shutdown_connections 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe081e170 0x55ebe0820560 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe080b8d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe080b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdcfbfff640 0 client.25529 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 --2- 172.21.15.23:0/3200999914 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 >> 172.21.15.23:0/3200999914 conn(0x55ebe07ff030 msgr2=0x55ebe0800d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 shutdown_connections 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 1 -- 172.21.15.23:0/3200999914 wait complete. 2024-03-28T15:54:00.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.050+0000 7fdd257a7740 20 client.25529 trim_cache size 0 max 16384 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 Processor -- start 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 -- start start 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1ae980 con 0x55ebe080de00 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1b0300 con 0x55ebe0805030 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.075+0000 7fdd257a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1aefe0 con 0x55ebe0804710 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd21079640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:42368/0 (socket says 172.21.15.23:42368) 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd20878640 1 -- 172.21.15.23:0/589609883 learned_addr learned my addr 172.21.15.23:0/589609883 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:48946/0 (socket says 172.21.15.23:48946) 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd21079640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45936/0 (socket says 172.21.15.23:45936) 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 -- 172.21.15.23:0/589609883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 msgr2=0x55ebe080d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 -- 172.21.15.23:0/589609883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 msgr2=0x55ebe08192c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 -- 172.21.15.23:0/589609883 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebe0809e00 con 0x55ebe0805030 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd20878640 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd21079640 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd080cca50 tx=0x7fdd08097b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ada3f71b18f7337 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd067fc640 1 -- 172.21.15.23:0/589609883 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd080b3020 con 0x55ebe0805030 2024-03-28T15:54:00.083 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd067fc640 1 -- 172.21.15.23:0/589609883 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd0809fa00 con 0x55ebe0805030 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.076+0000 7fdd067fc640 1 -- 172.21.15.23:0/589609883 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd080ccd00 con 0x55ebe0805030 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 -- 172.21.15.23:0/589609883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 msgr2=0x55ebe080b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd080cca50 tx=0x7fdd08097b50 comp rx=0 tx=0).stop 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 -- 172.21.15.23:0/589609883 shutdown_connections 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe080de00 0x55ebe08192c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe0805030 0x55ebe080b6a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 --2- 172.21.15.23:0/589609883 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe080d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 -- 172.21.15.23:0/589609883 >> 172.21.15.23:0/589609883 conn(0x55ebe07ff030 msgr2=0x55ebe07ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 -- 172.21.15.23:0/589609883 shutdown_connections 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.077+0000 7fdd257a7740 1 -- 172.21.15.23:0/589609883 wait complete. 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 -- start start 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 Processor -- start 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.084 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe0813850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1afca0 con 0x55ebe0805030 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1af640 con 0x55ebe080de00 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1c1acb40 con 0x55ebe0804710 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd257a7740 10 client.0 _ll_register_callbacks cb 0x7fdd140338d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd21079640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe0813850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd20878640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:42378/0 (socket says 172.21.15.23:42378) 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd1bfff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45944/0 (socket says 172.21.15.23:45944) 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.078+0000 7fdd20878640 1 -- 172.21.15.23:0/219859810 learned_addr learned my addr 172.21.15.23:0/219859810 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 -- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 msgr2=0x55ebe0813850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe0813850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.085 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 -- 172.21.15.23:0/219859810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 msgr2=0x55ebe0813310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 -- 172.21.15.23:0/219859810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebe0804e60 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd21079640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe0813850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd20878640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd1001e480 tx=0x7fdd10009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d100a2d5bef8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdcfbfff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd10038260 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd0809fa00 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fdd1001fa30 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd10048050 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd257a7740 10 client.25477 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.079+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ebe087edc0 con 0x55ebe0804710 2024-03-28T15:54:00.086 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd0809fa00 con 0x55ebe0804710 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fdd10005040 con 0x55ebe0804710 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 10 client.25477 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 10 client.25477 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 20 client.25477 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe087f1e0 0x55ebe0881600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 4 client.25477 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ebe07703a0 con 0x55ebe087f1e0 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe087f1e0 0x55ebe0881600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdcfbfff640 10 client.25477 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:00.087 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.080+0000 7fdd1bfff640 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe087f1e0 0x55ebe0881600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd993ac4f7613fbd server_cookie=c19ff53752bb5dff in_seq=0 out_seq=0 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdcfbfff640 1 -- 172.21.15.23:0/219859810 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ebe07703a0 con 0x55ebe087f1e0 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdcfbfff640 10 client.25477 handle_command_reply: tid=0 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 client.25477 shutdown 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 msgr2=0x55ebe0812dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd1001e480 tx=0x7fdd10009920 comp rx=0 tx=0).stop 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 shutdown_connections 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55ebe087f1e0 0x55ebe0881600 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ebe080de00 0x55ebe0813850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55ebe0805030 0x55ebe0813310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.089 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 --2- 172.21.15.23:0/219859810 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55ebe0804710 0x55ebe0812dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 >> 172.21.15.23:0/219859810 conn(0x55ebe07ff030 msgr2=0x55ebe080b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdcfbfff640 0 client.25477 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.083+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 shutdown_connections 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd257a7740 1 -- 172.21.15.23:0/219859810 wait complete. 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd257a7740 20 client.25477 trim_cache size 0 max 16384 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:54:00.090 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 msgr2=0x7fdd0c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 0x7fdd0c0a4f70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fdd10005fd0 tx=0x7fdd10009260 comp rx=0 tx=0).stop 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 msgr2=0x7fdd1c198490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdd0800e9f0 tx=0x7fdd0800edf0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 shutdown_connections 2024-03-28T15:54:00.091 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdd0c0a2ab0 0x7fdd0c0a4f70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd1c105540 0x7fdd1c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fdd1c1011b0 0x7fdd1c198490 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 --2- 172.21.15.23:0/3407798218 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fdd1c1007f0 0x7fdd1c197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 >> 172.21.15.23:0/3407798218 conn(0x7fdd1c0773f0 msgr2=0x7fdd1c0fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 shutdown_connections 2024-03-28T15:54:00.092 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:00.084+0000 7fdd22adc640 1 -- 172.21.15.23:0/3407798218 wait complete. 2024-03-28T15:54:00.577 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:00 smithi117 ceph-mon[36002]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:54:00.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:00 smithi120 ceph-mon[36826]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:54:00.647 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-mon[32448]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-28T15:54:00.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:54:00.644+0000 7ff0c040e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:54:00.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:54:00.644+0000 7ff0c040e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:54:00.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:54:00.644+0000 7ff0c040e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:54:00.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:54:00.644+0000 7ff0c040e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:54:00.921 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 15:54:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T15:54:00.644+0000 7ff0c040e700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-28T15:54:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:02 smithi120 ceph-mon[36826]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T15:54:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:02 smithi023 ceph-mon[32448]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T15:54:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:02 smithi117 ceph-mon[36002]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-03-28T15:54:03.420 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:03 smithi023 ceph-mon[32448]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-28T15:54:03.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:03 smithi117 ceph-mon[36002]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-28T15:54:03.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:03 smithi120 ceph-mon[36826]: mgrmap e24: x(active, since 57s), standbys: y, z 2024-03-28T15:54:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:04 smithi023 ceph-mon[32448]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:04 smithi117 ceph-mon[36002]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:04 smithi120 ceph-mon[36826]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:06 smithi023 ceph-mon[32448]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:54:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:06 smithi117 ceph-mon[36002]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:54:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:06 smithi120 ceph-mon[36826]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:54:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:08 smithi023 ceph-mon[32448]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:08 smithi117 ceph-mon[36002]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:08 smithi120 ceph-mon[36826]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:10 smithi023 ceph-mon[32448]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:10.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:10 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:10 smithi117 ceph-mon[36002]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:10 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:10 smithi120 ceph-mon[36826]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 24 op/s 2024-03-28T15:54:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:10 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:54:11.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:11.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:11 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:12.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:12.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:12.072 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:11 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:12.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 23 KiB/s wr, 28 op/s 2024-03-28T15:54:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:54:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:54:12.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:11 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:54:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 -- 172.21.15.23:0/156006779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc105540 msgr2=0x7f0adc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.221 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 --2- 172.21.15.23:0/156006779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc105540 0x7f0adc105920 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0acc009900 tx=0x7f0acc02bc30 comp rx=0 tx=0).stop 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 -- 172.21.15.23:0/156006779 shutdown_connections 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 --2- 172.21.15.23:0/156006779 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 0x7f0adc10d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 --2- 172.21.15.23:0/156006779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc1007f0 0x7f0adc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 --2- 172.21.15.23:0/156006779 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc105540 0x7f0adc105920 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 -- 172.21.15.23:0/156006779 >> 172.21.15.23:0/156006779 conn(0x7f0adc0773f0 msgr2=0x7f0adc077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 -- 172.21.15.23:0/156006779 shutdown_connections 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.219+0000 7f0ae169a640 1 -- 172.21.15.23:0/156006779 wait complete. 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.220+0000 7f0ae169a640 1 Processor -- start 2024-03-28T15:54:13.222 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.220+0000 7f0ae169a640 1 -- start start 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 0x7f0adc068da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc105540 0x7f0adc0692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc06bad0 con 0x7f0adc1007f0 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc06b950 con 0x7f0adc105540 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ae169a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc06bc50 con 0x7f0adc1011b0 2024-03-28T15:54:13.224 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41158/0 (socket says 172.21.15.23:41158) 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 -- 172.21.15.23:0/2002377821 learned_addr learned my addr 172.21.15.23:0/2002377821 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 0x7f0adc068da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc105540 0x7f0adc0692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 -- 172.21.15.23:0/2002377821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 msgr2=0x7f0adc068da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 0x7f0adc068da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 -- 172.21.15.23:0/2002377821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc105540 msgr2=0x7f0adc0692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc105540 0x7f0adc0692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 -- 172.21.15.23:0/2002377821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc069b10 con 0x7f0adc1007f0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.221+0000 7f0adaffd640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005940 tx=0x7f0acc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.222+0000 7f0ac3fff640 1 -- 172.21.15.23:0/2002377821 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc041070 con 0x7f0adc1007f0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.222+0000 7f0ac3fff640 1 -- 172.21.15.23:0/2002377821 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0acc0054e0 con 0x7f0adc1007f0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.222+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0adc1acd70 con 0x7f0adc1007f0 2024-03-28T15:54:13.225 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.222+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0adc1ad1b0 con 0x7f0adc1007f0 2024-03-28T15:54:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.223+0000 7f0ac3fff640 1 -- 172.21.15.23:0/2002377821 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0acc003d00 con 0x7f0adc1007f0 2024-03-28T15:54:13.226 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.225+0000 7f0ae4363740 1 Processor -- start 2024-03-28T15:54:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.225+0000 7f0ac3fff640 1 -- 172.21.15.23:0/2002377821 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 24) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f0acc044070 con 0x7f0adc1007f0 2024-03-28T15:54:13.227 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.225+0000 7f0ae4363740 1 -- start start 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe6caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 0x5606cfe758c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfdd0b10 con 0x5606cfe6c710 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfd79900 con 0x5606cfe6d030 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ae4363740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfdccbb0 con 0x5606cfe75e00 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0adb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe6caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.228 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:50020/0 (socket says 172.21.15.23:50020) 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 -- 172.21.15.23:0/2809661441 learned_addr learned my addr 172.21.15.23:0/2809661441 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0adaffd640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 0x5606cfe758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ac3fff640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 0x7f0ac40a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 -- 172.21.15.23:0/2809661441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 msgr2=0x5606cfe758c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 0x5606cfe758c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 -- 172.21.15.23:0/2809661441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 msgr2=0x5606cfe6caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe6caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ada7fc640 1 -- 172.21.15.23:0/2809661441 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606cfae5210 con 0x5606cfe75e00 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0adaffd640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 0x5606cfe758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:54:13.229 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe6caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.226+0000 7f0ac3fff640 1 -- 172.21.15.23:0/2002377821 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f0acc0f0800 con 0x7f0adc1007f0 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.227+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009c60 tx=0x7f0ad0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12df7c71d148a3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.227+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 0x7f0ac40a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.227+0000 7f0ac17fa640 1 -- 172.21.15.23:0/2809661441 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0004230 con 0x5606cfe75e00 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.227+0000 7f0ac17fa640 1 -- 172.21.15.23:0/2809661441 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ad0009e10 con 0x5606cfe75e00 2024-03-28T15:54:13.230 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.227+0000 7f0ac17fa640 1 -- 172.21.15.23:0/2809661441 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0019840 con 0x5606cfe75e00 2024-03-28T15:54:13.233 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- 172.21.15.23:0/2809661441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 msgr2=0x5606cfe781f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 0x7f0ac40a4ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4005fd0 tx=0x7f0ad4009ba0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ad0009c60 tx=0x7f0ad0011810 comp rx=0 tx=0).stop 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- 172.21.15.23:0/2809661441 shutdown_connections 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe781f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6d030 0x5606cfe758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 --2- 172.21.15.23:0/2809661441 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe6caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- 172.21.15.23:0/2809661441 >> 172.21.15.23:0/2809661441 conn(0x5606cfe67030 msgr2=0x5606cfe67410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- 172.21.15.23:0/2809661441 shutdown_connections 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- 172.21.15.23:0/2809661441 wait complete. 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 -- start start 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.228+0000 7f0ae4363740 1 Processor -- start 2024-03-28T15:54:13.234 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe72dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe72dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe72dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:39952/0 (socket says 172.21.15.23:39952) 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41182/0 (socket says 172.21.15.23:41182) 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0adaffd640 1 -- 172.21.15.23:0/4025577713 learned_addr learned my addr 172.21.15.23:0/4025577713 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe73840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfe78a30 con 0x5606cfe6d030 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfd8cd50 con 0x5606cfe6c710 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606cfe78bb0 con 0x5606cfe75e00 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.229+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe73840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 -- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 msgr2=0x5606cfe73840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe73840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.235 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 -- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 msgr2=0x5606cfe72dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe72dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606cfbb1620 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ae4363740 10 client.0 _ll_register_callbacks cb 0x5606cfd8c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ad0009e10 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x5606cfe68450 tx=0x7f0ad00052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8363a06475e21cde server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.230+0000 7f0abeff5640 10 client.15714 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0abeff5640 1 -- 172.21.15.23:0/4025577713 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0028020 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0abeff5640 1 -- 172.21.15.23:0/4025577713 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f0ad00035f0 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0abeff5640 1 -- 172.21.15.23:0/4025577713 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ad001f040 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 10 client.15714 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x5606cfe84bb0 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0abeff5640 1 -- 172.21.15.23:0/4025577713 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0ad0018070 con 0x5606cfe6d030 2024-03-28T15:54:13.236 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 10 client.15714 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 10 client.15714 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 20 client.15714 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfe85f00 0x5606cfe882f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.231+0000 7f0ae4363740 4 client.15714 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.232+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606cfdde050 con 0x5606cfe85f00 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.232+0000 7f0adaffd640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfe85f00 0x5606cfe882f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.232+0000 7f0abeff5640 10 client.15714 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:13.237 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.232+0000 7f0adaffd640 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfe85f00 0x5606cfe882f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2e2e7d3523b0252 server_cookie=75b558c9d2dcc960 in_seq=0 out_seq=0 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0abeff5640 1 -- 172.21.15.23:0/4025577713 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5606cfdde050 con 0x5606cfe85f00 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0abeff5640 10 client.15714 handle_command_reply: tid=0 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 client.15714 shutdown 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 msgr2=0x5606cfe73300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x5606cfe68450 tx=0x7f0ad00052f0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.238 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 shutdown_connections 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfe85f00 0x5606cfe882f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0abeff5640 0 client.15714 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe73840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6d030 0x5606cfe73300 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 --2- 172.21.15.23:0/4025577713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe72dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 >> 172.21.15.23:0/4025577713 conn(0x5606cfe67030 msgr2=0x5606cfe68d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 shutdown_connections 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.233+0000 7f0ae4363740 1 -- 172.21.15.23:0/4025577713 wait complete. 2024-03-28T15:54:13.239 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.234+0000 7f0ae4363740 20 client.15714 trim_cache size 0 max 16384 2024-03-28T15:54:13.262 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 Processor -- start 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 -- start start 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe73660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe68b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 0x5606cfe812c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe68b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe68b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:50040/0 (socket says 172.21.15.23:50040) 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ada7fc640 1 -- 172.21.15.23:0/3308953308 learned_addr learned my addr 172.21.15.23:0/3308953308 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b4cb0 con 0x5606cfe690d0 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b6630 con 0x5606cfe6c710 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0adaffd640 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe73660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 0x5606cfe812c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.263 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.261+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b5310 con 0x5606cfe75e00 2024-03-28T15:54:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 -- 172.21.15.23:0/3308953308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 msgr2=0x5606cfe68b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe68b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 -- 172.21.15.23:0/3308953308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 msgr2=0x5606cfe73660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe73660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.265 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 -- 172.21.15.23:0/3308953308 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606cfe6e990 con 0x5606cfe690d0 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.262+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 0x5606cfe812c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400f790 tx=0x7f0ad4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eda81e1fba1fbb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.263+0000 7f0ac0ff9640 1 -- 172.21.15.23:0/3308953308 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad400fd40 con 0x5606cfe690d0 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.263+0000 7f0ac0ff9640 1 -- 172.21.15.23:0/3308953308 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ad4010830 con 0x5606cfe690d0 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.263+0000 7f0ac0ff9640 1 -- 172.21.15.23:0/3308953308 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad40053c0 con 0x5606cfe690d0 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 msgr2=0x5606cfe812c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 0x5606cfe812c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400f790 tx=0x7f0ad4013000 comp rx=0 tx=0).stop 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 shutdown_connections 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe690d0 0x5606cfe812c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe75e00 0x5606cfe68b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 --2- 172.21.15.23:0/3308953308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe6c710 0x5606cfe73660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.266 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.264+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 >> 172.21.15.23:0/3308953308 conn(0x5606cfe67030 msgr2=0x5606cfe68480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.265+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 shutdown_connections 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.265+0000 7f0ae4363740 1 -- 172.21.15.23:0/3308953308 wait complete. 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.265+0000 7f0ae4363740 1 -- start start 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.265+0000 7f0ae4363740 1 Processor -- start 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 0x5606cfe794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ae4363740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 0x5606cfe79f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.269 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ada7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:41198/0 (socket says 172.21.15.23:41198) 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 0x5606cfe794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0adaffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 0x5606cfe794a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:50044/0 (socket says 172.21.15.23:50044) 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ada7fc640 1 -- 172.21.15.23:0/280082908 learned_addr learned my addr 172.21.15.23:0/280082908 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 0x5606cfe79f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.266+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b5fd0 con 0x5606cfe6c710 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b5970 con 0x5606cfe75e00 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b2e70 con 0x5606cfe690d0 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 -- 172.21.15.23:0/280082908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 msgr2=0x5606cfe794a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 0x5606cfe794a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 -- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 msgr2=0x5606cfe79f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:13.270 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 0x5606cfe79f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606cfee0860 con 0x5606cfe6c710 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0adb7fe640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 0x5606cfe79f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0ada7fc640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00194f0 tx=0x7f0ad0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b97ea29eebd52a6c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.267+0000 7f0abeff5640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.268+0000 7f0abeff5640 1 -- 172.21.15.23:0/280082908 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0028040 con 0x5606cfe6c710 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.268+0000 7f0abeff5640 1 -- 172.21.15.23:0/280082908 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f0ad00053e0 con 0x5606cfe6c710 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.268+0000 7f0ae4363740 10 client.0 _ll_register_callbacks cb 0x7f0ad0009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.268+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ad4010830 con 0x5606cfe6c710 2024-03-28T15:54:13.271 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.268+0000 7f0abeff5640 1 -- 172.21.15.23:0/280082908 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ad0045050 con 0x5606cfe6c710 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.269+0000 7f0ae4363740 10 client.15720 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x5606cfee1800 con 0x5606cfe6c710 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0abeff5640 1 -- 172.21.15.23:0/280082908 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f0ad0018050 con 0x5606cfe6c710 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 10 client.15720 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 10 client.15720 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:13.272 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 20 client.15720 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfee1e80 0x5606cfee4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 4 client.15720 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5606cfdd83a0 con 0x5606cfee1e80 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.270+0000 7f0adaffd640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfee1e80 0x5606cfee4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.271+0000 7f0abeff5640 10 client.15720 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:13.273 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.271+0000 7f0adaffd640 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfee1e80 0x5606cfee4340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22eada7745aeb283 server_cookie=b86bd6ad7e09a103 in_seq=0 out_seq=0 2024-03-28T15:54:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.275+0000 7f0abeff5640 1 -- 172.21.15.23:0/280082908 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5606cfdd83a0 con 0x5606cfee1e80 2024-03-28T15:54:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.275+0000 7f0abeff5640 10 client.15720 handle_command_reply: tid=0 2024-03-28T15:54:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.275+0000 7f0ae4363740 1 client.15720 shutdown 2024-03-28T15:54:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.275+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 msgr2=0x5606cfe799e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.277 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.275+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00194f0 tx=0x7f0ad0005f90 comp rx=0 tx=0).stop 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 shutdown_connections 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5606cfee1e80 0x5606cfee4340 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5606cfe75e00 0x5606cfe79f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5606cfe6c710 0x5606cfe799e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0abeff5640 0 client.15720 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 --2- 172.21.15.23:0/280082908 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5606cfe690d0 0x5606cfe794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 >> 172.21.15.23:0/280082908 conn(0x5606cfe67030 msgr2=0x5606cfe65790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 shutdown_connections 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 1 -- 172.21.15.23:0/280082908 wait complete. 2024-03-28T15:54:13.278 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae4363740 20 client.15720 trim_cache size 0 max 16384 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.276+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 msgr2=0x7f0ac40a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.279 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 0x7f0ac40a4ee0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4005fd0 tx=0x7f0ad4009ba0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 msgr2=0x7f0adc0730d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005940 tx=0x7f0acc004270 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 shutdown_connections 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0ac40a2a20 0x7f0ac40a4ee0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0adc105540 0x7f0adc0692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0adc1011b0 0x7f0adc068da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 --2- 172.21.15.23:0/2002377821 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0adc1007f0 0x7f0adc0730d0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 >> 172.21.15.23:0/2002377821 conn(0x7f0adc0773f0 msgr2=0x7f0adc0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 shutdown_connections 2024-03-28T15:54:13.280 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:13.277+0000 7f0ae169a640 1 -- 172.21.15.23:0/2002377821 wait complete. 2024-03-28T15:54:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:14 smithi117 ceph-mon[36002]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:14 smithi120 ceph-mon[36826]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:14 smithi023 ceph-mon[32448]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:16 smithi117 ceph-mon[36002]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:16 smithi120 ceph-mon[36826]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:16 smithi023 ceph-mon[32448]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:54:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:18 smithi117 ceph-mon[36002]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T15:54:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:18 smithi120 ceph-mon[36826]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T15:54:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:18 smithi023 ceph-mon[32448]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 682 B/s wr, 13 op/s 2024-03-28T15:54:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:20 smithi117 ceph-mon[36002]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T15:54:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:20 smithi120 ceph-mon[36826]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T15:54:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:20 smithi023 ceph-mon[32448]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-28T15:54:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:22 smithi117 ceph-mon[36002]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T15:54:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:22 smithi120 ceph-mon[36826]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T15:54:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:22 smithi023 ceph-mon[32448]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 597 B/s wr, 13 op/s 2024-03-28T15:54:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:24 smithi117 ceph-mon[36002]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:24 smithi120 ceph-mon[36826]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:24 smithi023 ceph-mon[32448]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:26.416 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 -- 172.21.15.23:0/707030881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 msgr2=0x7fb5e4100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/707030881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4100190 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009900 tx=0x7fb5d002bc30 comp rx=0 tx=0).stop 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 -- 172.21.15.23:0/707030881 shutdown_connections 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/707030881 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e41068d0 0x7fb5e410e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/707030881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e4100760 0x7fb5e41061f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/707030881 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4100190 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 -- 172.21.15.23:0/707030881 >> 172.21.15.23:0/707030881 conn(0x7fb5e4076ac0 msgr2=0x7fb5e40fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 -- 172.21.15.23:0/707030881 shutdown_connections 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.415+0000 7fb5eaf62640 1 -- 172.21.15.23:0/707030881 wait complete. 2024-03-28T15:54:26.417 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 Processor -- start 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 -- start start 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e419c780 0x7fb5e419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4111a50 con 0x7fb5e40ffdb0 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41118d0 con 0x7fb5e41068d0 2024-03-28T15:54:26.419 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5eaf62640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e4111bd0 con 0x7fb5e419c780 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53632/0 (socket says 172.21.15.23:53632) 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47384/0 (socket says 172.21.15.23:47384) 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 -- 172.21.15.23:0/3950071323 learned_addr learned my addr 172.21.15.23:0/3950071323 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e419c780 0x7fb5e419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 -- 172.21.15.23:0/3950071323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e419c780 msgr2=0x7fb5e419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e419c780 0x7fb5e419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 -- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 msgr2=0x7fb5e4197f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.416+0000 7fb5e3fff640 1 -- 172.21.15.23:0/3950071323 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e419d270 con 0x7fb5e41068d0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005fd0 tx=0x7fb5d800dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5e1ffb640 1 -- 172.21.15.23:0/3950071323 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8017070 con 0x7fb5e41068d0 2024-03-28T15:54:26.420 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e419d560 con 0x7fb5e41068d0 2024-03-28T15:54:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5e1ffb640 1 -- 172.21.15.23:0/3950071323 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5d8009c30 con 0x7fb5e41068d0 2024-03-28T15:54:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5e1ffb640 1 -- 172.21.15.23:0/3950071323 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8010460 con 0x7fb5e41068d0 2024-03-28T15:54:26.421 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.417+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e4100fc0 con 0x7fb5e41068d0 2024-03-28T15:54:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.419+0000 7fb5e1ffb640 1 -- 172.21.15.23:0/3950071323 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fb5d8004020 con 0x7fb5e41068d0 2024-03-28T15:54:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.420+0000 7fb5e1ffb640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 0x7fb5d40a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.420+0000 7fb5e1ffb640 1 -- 172.21.15.23:0/3950071323 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fb5d80cf1f0 con 0x7fb5e41068d0 2024-03-28T15:54:26.422 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.420+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 0x7fb5d40a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.420+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 0x7fb5d40a5070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006010 tx=0x7fb5d0005b50 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 Processor -- start 2024-03-28T15:54:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 -- start start 2024-03-28T15:54:26.423 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 0x55b8ac30baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 0x55b8ac3148c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 0x55b8ac3171f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 0x55b8ac30baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 0x55b8ac30baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:53412/0 (socket says 172.21.15.23:53412) 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 0x55b8ac3148c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 0x55b8ac3148c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53646/0 (socket says 172.21.15.23:53646) 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e8cfe640 1 -- 172.21.15.23:0/3137122744 learned_addr learned my addr 172.21.15.23:0/3137122744 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac26fb10 con 0x55b8ac314e00 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac218900 con 0x55b8ac30c030 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac26bbb0 con 0x55b8ac30b710 2024-03-28T15:54:26.424 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.421+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 0x55b8ac3171f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 -- 172.21.15.23:0/3137122744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 msgr2=0x55b8ac30baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 0x55b8ac30baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 -- 172.21.15.23:0/3137122744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 msgr2=0x55b8ac3148c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 0x55b8ac3148c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 -- 172.21.15.23:0/3137122744 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8abf84210 con 0x55b8ac314e00 2024-03-28T15:54:26.425 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 0x55b8ac3171f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009e50 tx=0x7fb5dc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1069ea6a719294c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5ceffd640 1 -- 172.21.15.23:0/3137122744 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc003b70 con 0x55b8ac314e00 2024-03-28T15:54:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5ceffd640 1 -- 172.21.15.23:0/3137122744 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5dc015070 con 0x55b8ac314e00 2024-03-28T15:54:26.426 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.422+0000 7fb5ceffd640 1 -- 172.21.15.23:0/3137122744 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc0117b0 con 0x55b8ac314e00 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 msgr2=0x55b8ac3171f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 0x55b8ac3171f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc009e50 tx=0x7fb5dc0107a0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 shutdown_connections 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac314e00 0x55b8ac3171f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30c030 0x55b8ac3148c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.427 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/3137122744 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30b710 0x55b8ac30baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 >> 172.21.15.23:0/3137122744 conn(0x55b8ac306030 msgr2=0x55b8ac306410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.423+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 shutdown_connections 2024-03-28T15:54:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.424+0000 7fb5edc2d740 1 -- 172.21.15.23:0/3137122744 wait complete. 2024-03-28T15:54:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.424+0000 7fb5edc2d740 1 -- start start 2024-03-28T15:54:26.428 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.424+0000 7fb5edc2d740 1 Processor -- start 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 0x55b8ac3123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac314e00 0x55b8ac312910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53652/0 (socket says 172.21.15.23:53652) 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 0x55b8ac3123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.431 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 0x55b8ac3123d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47396/0 (socket says 172.21.15.23:47396) 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac317a30 con 0x55b8ac30c030 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.425+0000 7fb5e8cfe640 1 -- 172.21.15.23:0/1027937666 learned_addr learned my addr 172.21.15.23:0/1027937666 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac22bd50 con 0x55b8ac30b710 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8ac317bb0 con 0x55b8ac314e00 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac314e00 0x55b8ac312910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5edc2d740 10 client.0 _ll_register_callbacks cb 0x55b8ac22b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 -- 172.21.15.23:0/1027937666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac314e00 msgr2=0x55b8ac312910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac314e00 0x55b8ac312910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 -- 172.21.15.23:0/1027937666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 msgr2=0x55b8ac3123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 0x55b8ac3123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8ac050620 con 0x55b8ac30b710 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036040 tx=0x7fb5d002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8cf45b8bcacb18 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.426+0000 7fb5c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d004f070 con 0x55b8ac30b710 2024-03-28T15:54:26.432 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5dc015070 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fb5d0005130 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5d0005330 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 10 client.25571 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b8ac323aa0 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5dc015070 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb5d00345e0 con 0x55b8ac30b710 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 10 client.25571 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 10 client.25571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 20 client.25571 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac327350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 4 client.25571 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.427+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b8ac27d050 con 0x55b8ac324f60 2024-03-28T15:54:26.433 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.428+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac327350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.428+0000 7fb5c3fff640 10 client.25571 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.428+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac327350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=434b17386097a9cf server_cookie=dd76545891616062 in_seq=0 out_seq=0 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1027937666 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b8ac27d050 con 0x55b8ac324f60 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5c3fff640 10 client.25571 handle_command_reply: tid=0 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 client.25571 shutdown 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 msgr2=0x55b8ac311e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036040 tx=0x7fb5d002bc30 comp rx=0 tx=0).stop 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 shutdown_connections 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac327350 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac314e00 0x55b8ac312910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5c3fff640 0 client.25571 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:26.435 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30c030 0x55b8ac3123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1027937666 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac30b710 0x55b8ac311e90 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 >> 172.21.15.23:0/1027937666 conn(0x55b8ac306030 msgr2=0x55b8ac307d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 shutdown_connections 2024-03-28T15:54:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1027937666 wait complete. 2024-03-28T15:54:26.436 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.429+0000 7fb5edc2d740 20 client.25571 trim_cache size 0 max 16384 2024-03-28T15:54:26.465 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 Processor -- start 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 -- start start 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac3145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac3202c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41b0c20 con 0x55b8ac30b710 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41b25a0 con 0x55b8ac314e00 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41b1280 con 0x55b8ac30c030 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.443+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac3145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:53420/0 (socket says 172.21.15.23:53420) 2024-03-28T15:54:26.466 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 -- 172.21.15.23:0/1676558780 learned_addr learned my addr 172.21.15.23:0/1676558780 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac3202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 -- 172.21.15.23:0/1676558780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 msgr2=0x55b8ac3202c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac3202c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 -- 172.21.15.23:0/1676558780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 msgr2=0x55b8ac3145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac3145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 -- 172.21.15.23:0/1676558780 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b8ac305b60 con 0x55b8ac30c030 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac3202c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8097630 tx=0x7fb5d80975a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=768990eb28ee2869 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5ce7fc640 1 -- 172.21.15.23:0/1676558780 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d809f320 con 0x55b8ac30c030 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5ce7fc640 1 -- 172.21.15.23:0/1676558780 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5d80a3970 con 0x55b8ac30c030 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.444+0000 7fb5ce7fc640 1 -- 172.21.15.23:0/1676558780 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80bb140 con 0x55b8ac30c030 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1676558780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 msgr2=0x55b8ac307b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8097630 tx=0x7fb5d80975a0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.467 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1676558780 shutdown_connections 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac3202c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac307b90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1676558780 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac3145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1676558780 >> 172.21.15.23:0/1676558780 conn(0x55b8ac306030 msgr2=0x55b8ac307480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1676558780 shutdown_connections 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1676558780 wait complete. 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.445+0000 7fb5edc2d740 1 -- start start 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 Processor -- start 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac30ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41b1f40 con 0x55b8ac30b710 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41b18e0 con 0x55b8ac314e00 2024-03-28T15:54:26.468 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e41aede0 con 0x55b8ac30c030 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac30ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5edc2d740 10 client.0 _ll_register_callbacks cb 0x7fb5d002c990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e94ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e3fff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:53432/0 (socket says 172.21.15.23:53432) 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e3fff640 1 -- 172.21.15.23:0/1352058256 learned_addr learned my addr 172.21.15.23:0/1352058256 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e94ff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53674/0 (socket says 172.21.15.23:53674) 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.446+0000 7fb5e8cfe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac30ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:47422/0 (socket says 172.21.15.23:47422) 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 -- 172.21.15.23:0/1352058256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 msgr2=0x55b8ac30f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 -- 172.21.15.23:0/1352058256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 msgr2=0x55b8ac30ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac30ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.469 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 -- 172.21.15.23:0/1352058256 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b8ac382630 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e3fff640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5e94ff640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0049d0 tx=0x7fb5dc01c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe79579ba6ebfe42 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5c3fff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc009500 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7fb5dc01ca20 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5d80a3970 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5dc01cc20 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5edc2d740 10 client.25577 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.447+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b8ac3899e0 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5d80a3970 con 0x55b8ac314e00 2024-03-28T15:54:26.470 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7fb5dc005520 con 0x55b8ac314e00 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 10 client.25577 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 10 client.25577 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 20 client.25577 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac38be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 4 client.25577 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b8ac2773a0 con 0x55b8ac324f60 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.448+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac38be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.449+0000 7fb5c3fff640 10 client.25577 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:26.472 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.449+0000 7fb5e8cfe640 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac38be10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf2fa95c27741b8a server_cookie=7085352db0b96350 in_seq=0 out_seq=0 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5c3fff640 1 -- 172.21.15.23:0/1352058256 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b8ac2773a0 con 0x55b8ac324f60 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5c3fff640 10 client.25577 handle_command_reply: tid=0 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 client.25577 shutdown 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 msgr2=0x55b8ac381fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc0049d0 tx=0x7fb5dc01c6a0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 shutdown_connections 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b8ac324f60 0x55b8ac38be10 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5c3fff640 0 client.25577 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b8ac314e00 0x55b8ac381fc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.452+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b8ac30c030 0x55b8ac30f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.453+0000 7fb5edc2d740 1 --2- 172.21.15.23:0/1352058256 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b8ac30b710 0x55b8ac30ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.453+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 >> 172.21.15.23:0/1352058256 conn(0x55b8ac306030 msgr2=0x55b8ac305020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.453+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 shutdown_connections 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.453+0000 7fb5edc2d740 1 -- 172.21.15.23:0/1352058256 wait complete. 2024-03-28T15:54:26.473 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.453+0000 7fb5edc2d740 20 client.25577 trim_cache size 0 max 16384 2024-03-28T15:54:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:54:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:54:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:54:26.474 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 msgr2=0x7fb5d40a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 0x7fb5d40a5070 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0006010 tx=0x7fb5d0005b50 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 msgr2=0x7fb5e4198450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8005fd0 tx=0x7fb5d800dde0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 shutdown_connections 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb5d40a2bb0 0x7fb5d40a5070 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb5e419c780 0x7fb5e419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5e41068d0 0x7fb5e4198450 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 --2- 172.21.15.23:0/3950071323 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb5e40ffdb0 0x7fb5e4197f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 >> 172.21.15.23:0/3950071323 conn(0x7fb5e4076ac0 msgr2=0x7fb5e4105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 shutdown_connections 2024-03-28T15:54:26.475 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:26.454+0000 7fb5eaf62640 1 -- 172.21.15.23:0/3950071323 wait complete. 2024-03-28T15:54:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:26 smithi117 ceph-mon[36002]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:26 smithi120 ceph-mon[36826]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:26 smithi023 ceph-mon[32448]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-03-28T15:54:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:28 smithi117 ceph-mon[36002]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T15:54:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:28 smithi120 ceph-mon[36826]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T15:54:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:28 smithi023 ceph-mon[32448]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 511 B/s wr, 13 op/s 2024-03-28T15:54:30.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:30 smithi117 ceph-mon[36002]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:54:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:30 smithi120 ceph-mon[36826]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:54:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:30 smithi023 ceph-mon[32448]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:54:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:32 smithi117 ceph-mon[36002]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T15:54:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:32 smithi120 ceph-mon[36826]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T15:54:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:32 smithi023 ceph-mon[32448]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-28T15:54:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:34 smithi117 ceph-mon[36002]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:34 smithi120 ceph-mon[36826]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:34 smithi023 ceph-mon[32448]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:36 smithi117 ceph-mon[36002]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:36 smithi120 ceph-mon[36826]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:36 smithi023 ceph-mon[32448]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-28T15:54:37.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:37 smithi117 ceph-mon[36002]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-28T15:54:37.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:37 smithi120 ceph-mon[36826]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-28T15:54:37.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:37 smithi023 ceph-mon[32448]: mgrmap e25: x(active, since 91s), standbys: y, z 2024-03-28T15:54:38.207 INFO:tasks.workunit.client.0.smithi023.stderr:Updating files: 93% (11488/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr:state without impacting any branches by switching back to a branch. 2024-03-28T15:54:38.538 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: git switch -c 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr:Or undo this operation with: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: git switch - 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:38.539 INFO:tasks.workunit.client.0.smithi023.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-28T15:54:38.543 DEBUG:teuthology.orchestra.run.smithi023:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-28T15:54:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:38 smithi117 ceph-mon[36002]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-28T15:54:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:38 smithi120 ceph-mon[36826]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-28T15:54:38.609 INFO:tasks.workunit.client.0.smithi023.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-28T15:54:38.611 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T15:54:38.611 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-28T15:54:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:38 smithi023 ceph-mon[32448]: pgmap v49: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-03-28T15:54:39.300 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-28T15:54:39.357 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-28T15:54:39.396 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-28T15:54:39.398 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T15:54:39.398 INFO:tasks.workunit.client.0.smithi023.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-28T15:54:39.437 INFO:tasks.workunit.client.0.smithi023.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-28T15:54:39.440 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T15:54:39.441 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-28T15:54:39.497 INFO:tasks.workunit:Running workunits matching suites/fsync-tester.sh on client.0... 2024-03-28T15:54:39.499 INFO:tasks.workunit:Running workunit suites/fsync-tester.sh... 2024-03-28T15:54:39.499 DEBUG:teuthology.orchestra.run.smithi023:workunit test suites/fsync-tester.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 -- 172.21.15.23:0/4254680732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f20105540 msgr2=0x7f9f20105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 --2- 172.21.15.23:0/4254680732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f20105540 0x7f9f20105920 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9f18009ca0 tx=0x7f9f1802c0f0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 -- 172.21.15.23:0/4254680732 shutdown_connections 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 --2- 172.21.15.23:0/4254680732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f201011b0 0x7f9f2010d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 --2- 172.21.15.23:0/4254680732 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 0x7f9f20100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 --2- 172.21.15.23:0/4254680732 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f20105540 0x7f9f20105920 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.552 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 -- 172.21.15.23:0/4254680732 >> 172.21.15.23:0/4254680732 conn(0x7f9f200773f0 msgr2=0x7f9f20077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 -- 172.21.15.23:0/4254680732 shutdown_connections 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.550+0000 7f9f26b22640 1 -- 172.21.15.23:0/4254680732 wait complete. 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 Processor -- start 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 -- start start 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 0x7f9f20197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f20105540 0x7f9f2019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20110a60 con 0x7f9f201011b0 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201108e0 con 0x7f9f20105540 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f26b22640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20110be0 con 0x7f9f201007f0 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f248be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 0x7f9f20197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.553 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54470/0 (socket says 172.21.15.23:54470) 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 -- 172.21.15.23:0/108711130 learned_addr learned my addr 172.21.15.23:0/108711130 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f250bf640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f20105540 0x7f9f2019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 -- 172.21.15.23:0/108711130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 msgr2=0x7f9f20197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 0x7f9f20197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 -- 172.21.15.23:0/108711130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f20105540 msgr2=0x7f9f2019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f20105540 0x7f9f2019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 -- 172.21.15.23:0/108711130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2019cf80 con 0x7f9f201011b0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.551+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000e9f0 tx=0x7f9f1000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.552+0000 7f9f1dffb640 1 -- 172.21.15.23:0/108711130 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1000cd80 con 0x7f9f201011b0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.552+0000 7f9f1dffb640 1 -- 172.21.15.23:0/108711130 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f10005500 con 0x7f9f201011b0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.552+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f2019d270 con 0x7f9f201011b0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.552+0000 7f9f1dffb640 1 -- 172.21.15.23:0/108711130 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10010690 con 0x7f9f201011b0 2024-03-28T15:54:39.554 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.552+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f201a4b00 con 0x7f9f201011b0 2024-03-28T15:54:39.559 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.554+0000 7f9f1dffb640 1 -- 172.21.15.23:0/108711130 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9f100046c0 con 0x7f9f201011b0 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.554+0000 7f9f297ee740 1 Processor -- start 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.554+0000 7f9f297ee740 1 -- start start 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4876af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a487fe00 0x5572a48821f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a47dab10 con 0x5572a487fe00 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a4783900 con 0x5572a4877030 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f297ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a47d6bb0 con 0x5572a4876710 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f1dffb640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 0x7f9f0c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f250bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4876af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a487fe00 0x5572a48821f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53276/0 (socket says 172.21.15.23:53276) 2024-03-28T15:54:39.560 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f250bf640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4876af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:44060/0 (socket says 172.21.15.23:44060) 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 -- 172.21.15.23:0/1511225522 learned_addr learned my addr 172.21.15.23:0/1511225522 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f250bf640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 0x7f9f0c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 -- 172.21.15.23:0/1511225522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 msgr2=0x5572a4876af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4876af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 -- 172.21.15.23:0/1511225522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a487fe00 msgr2=0x5572a48821f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a487fe00 0x5572a48821f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 -- 172.21.15.23:0/1511225522 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572a44ef210 con 0x5572a4877030 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f1dffb640 1 -- 172.21.15.23:0/108711130 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f9f100cf620 con 0x7f9f201011b0 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.555+0000 7f9f248be640 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802c700 tx=0x7f9f180098c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0a31729e401a075 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f250bf640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 0x7f9f0c0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009660 tx=0x7f9f08004730 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f16ffd640 1 -- 172.21.15.23:0/1511225522 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18043020 con 0x5572a4877030 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f16ffd640 1 -- 172.21.15.23:0/1511225522 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f18003970 con 0x5572a4877030 2024-03-28T15:54:39.561 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f16ffd640 1 -- 172.21.15.23:0/1511225522 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18039040 con 0x5572a4877030 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 -- 172.21.15.23:0/1511225522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 msgr2=0x5572a487f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802c700 tx=0x7f9f180098c0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 -- 172.21.15.23:0/1511225522 shutdown_connections 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a487fe00 0x5572a48821f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a487f8c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1511225522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4876af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.556+0000 7f9f297ee740 1 -- 172.21.15.23:0/1511225522 >> 172.21.15.23:0/1511225522 conn(0x5572a4871030 msgr2=0x5572a4871410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- 172.21.15.23:0/1511225522 shutdown_connections 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- 172.21.15.23:0/1511225522 wait complete. 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- start start 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 Processor -- start 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a487ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.562 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a487d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a48829d0 con 0x5572a4877030 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a4796d50 con 0x5572a487fe00 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572a4882b50 con 0x5572a4876710 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.557+0000 7f9f297ee740 10 client.0 _ll_register_callbacks cb 0x5572a4796720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a487d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f248be640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a487ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:53288/0 (socket says 172.21.15.23:53288) 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 -- 172.21.15.23:0/4089053793 learned_addr learned my addr 172.21.15.23:0/4089053793 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 -- 172.21.15.23:0/4089053793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 msgr2=0x5572a487ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a487ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 -- 172.21.15.23:0/4089053793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 msgr2=0x5572a487d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.563 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a487d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 -- 172.21.15.23:0/4089053793 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572a45bb620 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f248be640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a487ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a487d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f250bf640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f08012420 tx=0x7f9f080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5883dbc40f94340c server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9efffff640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08010640 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f9f080107e0 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.558+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f18003970 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f08030050 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 10 client.25589 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5572a488ec90 con 0x5572a487fe00 2024-03-28T15:54:39.564 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f18003970 con 0x5572a487fe00 2024-03-28T15:54:39.565 INFO:tasks.workunit.client.0.smithi023.stderr:+ wget http://download.ceph.com/qa/fsync-tester.c 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f9f08022af0 con 0x5572a487fe00 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 10 client.25589 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 10 client.25589 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 20 client.25589 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a4892540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 4 client.25589 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572a47e8050 con 0x5572a4890150 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.559+0000 7f9f248be640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a4892540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f248be640 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a4892540 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f336249102609086 server_cookie=bafbb3f2d9da2a3b in_seq=0 out_seq=0 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9efffff640 10 client.25589 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9efffff640 1 -- 172.21.15.23:0/4089053793 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5572a47e8050 con 0x5572a4890150 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9efffff640 10 client.25589 handle_command_reply: tid=0 2024-03-28T15:54:39.567 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 client.25589 shutdown 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 msgr2=0x5572a487d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f08012420 tx=0x7f9f080046c0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 shutdown_connections 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a4892540 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a487d8b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9efffff640 0 client.25589 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a487d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 --2- 172.21.15.23:0/4089053793 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a487ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.560+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 >> 172.21.15.23:0/4089053793 conn(0x5572a4871030 msgr2=0x5572a4872d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.561+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 shutdown_connections 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.561+0000 7f9f297ee740 1 -- 172.21.15.23:0/4089053793 wait complete. 2024-03-28T15:54:39.568 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.561+0000 7f9f297ee740 20 client.25589 trim_cache size 0 max 16384 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 Processor -- start 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 -- start start 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a488b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a487fe00 0x5572a488b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201ae980 con 0x5572a4876710 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201b0300 con 0x5572a4877030 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f297ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201aefe0 con 0x5572a487fe00 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.577 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54492/0 (socket says 172.21.15.23:54492) 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 -- 172.21.15.23:0/2643423416 learned_addr learned my addr 172.21.15.23:0/2643423416 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f250bf640 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a487fe00 0x5572a488b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a488b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 -- 172.21.15.23:0/2643423416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a487fe00 msgr2=0x5572a488b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a487fe00 0x5572a488b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 -- 172.21.15.23:0/2643423416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 msgr2=0x5572a488b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a488b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 -- 172.21.15.23:0/2643423416 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572a488bce0 con 0x5572a4876710 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.575+0000 7f9f248be640 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802c700 tx=0x7f9f18000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43c9ccbfdc57338 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f167fc640 1 -- 172.21.15.23:0/2643423416 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18058070 con 0x5572a4876710 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f167fc640 1 -- 172.21.15.23:0/2643423416 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f18003970 con 0x5572a4876710 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f167fc640 1 -- 172.21.15.23:0/2643423416 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f18051070 con 0x5572a4876710 2024-03-28T15:54:39.578 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 -- 172.21.15.23:0/2643423416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 msgr2=0x5572a487f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f9f1802c700 tx=0x7f9f18000f40 comp rx=0 tx=0).stop 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 -- 172.21.15.23:0/2643423416 shutdown_connections 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a487fe00 0x5572a488b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a4877030 0x5572a488b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 --2- 172.21.15.23:0/2643423416 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4876710 0x5572a487f560 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 -- 172.21.15.23:0/2643423416 >> 172.21.15.23:0/2643423416 conn(0x5572a4871030 msgr2=0x5572a4871410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 -- 172.21.15.23:0/2643423416 shutdown_connections 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.576+0000 7f9f297ee740 1 -- 172.21.15.23:0/2643423416 wait complete. 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 -- start start 2024-03-28T15:54:39.579 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 Processor -- start 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4883320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a4883da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201afca0 con 0x5572a4877030 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201af640 con 0x5572a487fe00 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201acb40 con 0x5572a4876710 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f297ee740 10 client.0 _ll_register_callbacks cb 0x7f9f08004100 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.582 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f1ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54500/0 (socket says 172.21.15.23:54500) 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f1ffff640 1 -- 172.21.15.23:0/1359178495 learned_addr learned my addr 172.21.15.23:0/1359178495 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f248be640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4883320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.577+0000 7f9f250bf640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a4883da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 -- 172.21.15.23:0/1359178495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 msgr2=0x5572a4883320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4883320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 -- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 msgr2=0x5572a4883da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a4883da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 -- 172.21.15.23:0/1359178495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572a48eaa10 con 0x5572a4877030 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f1ffff640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9f10097450 tx=0x7f9f100d1b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6385a6d62bd758 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1009bfb0 con 0x5572a4877030 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f9f100860b0 con 0x5572a4877030 2024-03-28T15:54:39.583 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9f18003970 con 0x5572a4877030 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f100862b0 con 0x5572a4877030 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 10 client.15756 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5572a48ebd00 con 0x5572a4877030 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f18003970 con 0x5572a4877030 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f9f100a3bc0 con 0x5572a4877030 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 10 client.15756 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 10 client.15756 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 20 client.15756 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a48ee470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 4 client.15756 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:54:39.584 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.578+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5572a47e23a0 con 0x5572a4890150 2024-03-28T15:54:39.585 INFO:tasks.workunit.client.0.smithi023.stderr:--2024-03-28 15:54:39-- http://download.ceph.com/qa/fsync-tester.c 2024-03-28T15:54:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.579+0000 7f9f248be640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a48ee470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.579+0000 7f9efffff640 10 client.15756 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:39.585 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.579+0000 7f9f248be640 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a48ee470 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5c765e814cbf4b6 server_cookie=2f1f97b249c2e4bc in_seq=0 out_seq=0 2024-03-28T15:54:39.586 INFO:tasks.workunit.client.0.smithi023.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-28T15:54:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.586+0000 7f9efffff640 1 -- 172.21.15.23:0/1359178495 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5572a47e23a0 con 0x5572a4890150 2024-03-28T15:54:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.586+0000 7f9efffff640 10 client.15756 handle_command_reply: tid=0 2024-03-28T15:54:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.586+0000 7f9f297ee740 1 client.15756 shutdown 2024-03-28T15:54:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 msgr2=0x5572a4883860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.587 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f9f10097450 tx=0x7f9f100d1b30 comp rx=0 tx=0).stop 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 shutdown_connections 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5572a4890150 0x5572a48ee470 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572a487fe00 0x5572a4883da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5572a4877030 0x5572a4883860 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9efffff640 0 client.15756 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 --2- 172.21.15.23:0/1359178495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5572a4876710 0x5572a4883320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 >> 172.21.15.23:0/1359178495 conn(0x5572a4871030 msgr2=0x5572a486f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 shutdown_connections 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 1 -- 172.21.15.23:0/1359178495 wait complete. 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f297ee740 20 client.15756 trim_cache size 0 max 16384 2024-03-28T15:54:39.588 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:54:39.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:54:39.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:54:39.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:54:39.589 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 msgr2=0x7f9f0c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 0x7f9f0c0a4ed0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9f08009660 tx=0x7f9f08004730 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 msgr2=0x7f9f20198520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000e9f0 tx=0x7f9f1000edf0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 shutdown_connections 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9f0c0a2a10 0x7f9f0c0a4ed0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f20105540 0x7f9f2019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9f201011b0 0x7f9f20198520 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 --2- 172.21.15.23:0/108711130 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9f201007f0 0x7f9f20197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.587+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 >> 172.21.15.23:0/108711130 conn(0x7f9f200773f0 msgr2=0x7f9f20068fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.588+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 shutdown_connections 2024-03-28T15:54:39.590 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:39.588+0000 7f9f26b22640 1 -- 172.21.15.23:0/108711130 wait complete. 2024-03-28T15:54:39.648 INFO:tasks.workunit.client.0.smithi023.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-28T15:54:39.703 INFO:tasks.workunit.client.0.smithi023.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-28T15:54:39.703 INFO:tasks.workunit.client.0.smithi023.stderr:Length: 3024 (3.0K) [application/octet-stream] 2024-03-28T15:54:39.705 INFO:tasks.workunit.client.0.smithi023.stderr:Saving to: ‘fsync-tester.c’ 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr: 0K .. 100% 604M=0s 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr:2024-03-28 15:54:39 (604 MB/s) - ‘fsync-tester.c’ saved [3024/3024] 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr: 2024-03-28T15:54:39.709 INFO:tasks.workunit.client.0.smithi023.stderr:+ gcc -D_GNU_SOURCE fsync-tester.c -o fsync-tester 2024-03-28T15:54:39.777 INFO:tasks.workunit.client.0.smithi023.stderr:+ ./fsync-tester 2024-03-28T15:54:40.574 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:40 smithi117 ceph-mon[36002]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:54:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:40 smithi120 ceph-mon[36826]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:54:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:40 smithi023 ceph-mon[32448]: pgmap v50: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-28T15:54:42.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:42 smithi117 ceph-mon[36002]: pgmap v51: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 179 KiB/s wr, 18 op/s 2024-03-28T15:54:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:42 smithi120 ceph-mon[36826]: pgmap v51: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 179 KiB/s wr, 18 op/s 2024-03-28T15:54:42.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:42 smithi023 ceph-mon[32448]: pgmap v51: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 179 KiB/s wr, 18 op/s 2024-03-28T15:54:44.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:44 smithi117 ceph-mon[36002]: pgmap v52: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:44 smithi120 ceph-mon[36826]: pgmap v52: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:44.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:44 smithi023 ceph-mon[32448]: pgmap v52: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:46.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:46 smithi117 ceph-mon[36002]: pgmap v53: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:46 smithi120 ceph-mon[36826]: pgmap v53: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:46 smithi023 ceph-mon[32448]: pgmap v53: 129 pgs: 129 active+clean; 2.4 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 179 KiB/s wr, 14 op/s 2024-03-28T15:54:48.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:48 smithi117 ceph-mon[36002]: pgmap v54: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 610 KiB/s wr, 19 op/s 2024-03-28T15:54:48.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:48 smithi120 ceph-mon[36826]: pgmap v54: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 610 KiB/s wr, 19 op/s 2024-03-28T15:54:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:48 smithi023 ceph-mon[32448]: pgmap v54: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 610 KiB/s wr, 19 op/s 2024-03-28T15:54:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:50 smithi117 ceph-mon[36002]: pgmap v55: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 609 KiB/s wr, 14 op/s 2024-03-28T15:54:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:50 smithi120 ceph-mon[36826]: pgmap v55: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 609 KiB/s wr, 14 op/s 2024-03-28T15:54:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:50 smithi023 ceph-mon[32448]: pgmap v55: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 609 KiB/s wr, 14 op/s 2024-03-28T15:54:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:52 smithi117 ceph-mon[36002]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-03-28T15:54:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:52 smithi120 ceph-mon[36826]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-03-28T15:54:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:52 smithi023 ceph-mon[32448]: pgmap v56: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.0 MiB/s wr, 19 op/s 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 -- 172.21.15.23:0/2432498504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec106660 msgr2=0x7ffbec106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 --2- 172.21.15.23:0/2432498504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec106660 0x7ffbec106a40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc009900 tx=0x7ffbdc02bc30 comp rx=0 tx=0).stop 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 -- 172.21.15.23:0/2432498504 shutdown_connections 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 --2- 172.21.15.23:0/2432498504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec100f60 0x7ffbec10d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 --2- 172.21.15.23:0/2432498504 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec1005a0 0x7ffbec100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 --2- 172.21.15.23:0/2432498504 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec106660 0x7ffbec106a40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 -- 172.21.15.23:0/2432498504 >> 172.21.15.23:0/2432498504 conn(0x7ffbec077700 msgr2=0x7ffbec077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 -- 172.21.15.23:0/2432498504 shutdown_connections 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.677+0000 7ffbf1012640 1 -- 172.21.15.23:0/2432498504 wait complete. 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.678+0000 7ffbf1012640 1 Processor -- start 2024-03-28T15:54:52.686 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.678+0000 7ffbf1012640 1 -- start start 2024-03-28T15:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec100f60 0x7ffbec198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.687 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 0x7ffbec19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.695 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec110960 con 0x7ffbec1005a0 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1107e0 con 0x7ffbec106660 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbf1012640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec110ae0 con 0x7ffbec100f60 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45282/0 (socket says 172.21.15.23:45282) 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 -- 172.21.15.23:0/591377936 learned_addr learned my addr 172.21.15.23:0/591377936 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbeb59e640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 0x7ffbec19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbea59c640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec100f60 0x7ffbec198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 -- 172.21.15.23:0/591377936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec100f60 msgr2=0x7ffbec198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec100f60 0x7ffbec198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 -- 172.21.15.23:0/591377936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 msgr2=0x7ffbec19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 0x7ffbec19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 -- 172.21.15.23:0/591377936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec19ce20 con 0x7ffbec1005a0 2024-03-28T15:54:52.696 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbeb59e640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 0x7ffbec19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbead9d640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005940 tx=0x7ffbdc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.679+0000 7ffbd3fff640 1 -- 172.21.15.23:0/591377936 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc041070 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.680+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec19d0b0 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.680+0000 7ffbd3fff640 1 -- 172.21.15.23:0/591377936 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc0054e0 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.680+0000 7ffbd3fff640 1 -- 172.21.15.23:0/591377936 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc003d00 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.680+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec1a49a0 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.682+0000 7ffbd3fff640 1 -- 172.21.15.23:0/591377936 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ffbdc044070 con 0x7ffbec1005a0 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.682+0000 7ffbf3bff740 1 Processor -- start 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.682+0000 7ffbf3bff740 1 -- start start 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.682+0000 7ffbd3fff640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 0x7ffbd40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.697 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb32cb10 con 0x55b4fb3d1e00 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb2d5900 con 0x55b4fb3c9030 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb328bb0 con 0x55b4fb3c8710 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbd3fff640 1 -- 172.21.15.23:0/591377936 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ffbdc0f0d60 con 0x7ffbec1005a0 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbeb59e640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 0x7ffbd40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbeb59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:33276/0 (socket says 172.21.15.23:33276) 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 -- 172.21.15.23:0/3660454207 learned_addr learned my addr 172.21.15.23:0/3660454207 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbea59c640 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 -- 172.21.15.23:0/3660454207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 msgr2=0x55b4fb3d18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 -- 172.21.15.23:0/3660454207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 msgr2=0x55b4fb3d41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.698 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 -- 172.21.15.23:0/3660454207 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4fb041210 con 0x55b4fb3c8710 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbeb59e640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 0x7ffbd40a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009fd0 tx=0x7ffbe40116b0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbead9d640 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0c5370 tx=0x7ffbdc0b9510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4990279907f00f48 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.683+0000 7ffbd17fa640 1 -- 172.21.15.23:0/3660454207 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc04b040 con 0x55b4fb3c8710 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbd17fa640 1 -- 172.21.15.23:0/3660454207 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc005690 con 0x55b4fb3c8710 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbd17fa640 1 -- 172.21.15.23:0/3660454207 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0dd070 con 0x55b4fb3c8710 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 -- 172.21.15.23:0/3660454207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 msgr2=0x55b4fb3c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0c5370 tx=0x7ffbdc0b9510 comp rx=0 tx=0).stop 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 -- 172.21.15.23:0/3660454207 shutdown_connections 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/3660454207 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3c8af0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 -- 172.21.15.23:0/3660454207 >> 172.21.15.23:0/3660454207 conn(0x55b4fb3c3030 msgr2=0x55b4fb3c3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.699 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 -- 172.21.15.23:0/3660454207 shutdown_connections 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.684+0000 7ffbf3bff740 1 -- 172.21.15.23:0/3660454207 wait complete. 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 -- start start 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 Processor -- start 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb328bb0 con 0x55b4fb3c8710 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb2e8d50 con 0x55b4fb3d1e00 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4fb3d47f0 con 0x55b4fb3c9030 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbf3bff740 10 client.0 _ll_register_callbacks cb 0x55b4fb2e8720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:45298/0 (socket says 172.21.15.23:45298) 2024-03-28T15:54:52.700 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbead9d640 1 -- 172.21.15.23:0/2466903182 learned_addr learned my addr 172.21.15.23:0/2466903182 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbeb59e640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 -- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3d1e00 msgr2=0x55b4fb3cf920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cf920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 -- 172.21.15.23:0/2466903182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 msgr2=0x55b4fb3ceea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbea59c640 1 -- 172.21.15.23:0/2466903182 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4fb10d620 con 0x55b4fb3c9030 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.685+0000 7ffbead9d640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cf3e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffbec100300 tx=0x7ffbe0012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77eef3d671db02c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe000f8c0 con 0x55b4fb3c9030 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b4fb33a050 con 0x55b4fb3c9030 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ffbe000fa60 con 0x55b4fb3c9030 2024-03-28T15:54:52.701 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbe000fc60 con 0x55b4fb3c9030 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 10 client.25549 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b4fb3e09e0 con 0x55b4fb3c9030 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b4fb33a050 con 0x55b4fb3c9030 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ffbe001c420 con 0x55b4fb3c9030 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 10 client.25549 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 10 client.25549 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 20 client.25549 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb3e4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 4 client.25549 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.686+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4fb3343a0 con 0x55b4fb3e1e50 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.687+0000 7ffbead9d640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb3e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.687+0000 7ffbead9d640 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb3e4240 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0033b052825f3e4 server_cookie=50a001b92f2383c6 in_seq=0 out_seq=0 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.687+0000 7ffbceff5640 10 client.25549 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:52.702 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbceff5640 1 -- 172.21.15.23:0/2466903182 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b4fb3343a0 con 0x55b4fb3e1e50 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbceff5640 10 client.25549 handle_command_reply: tid=0 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 client.25549 shutdown 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 msgr2=0x55b4fb3cf3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cf3e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffbec100300 tx=0x7ffbe0012b60 comp rx=0 tx=0).stop 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 shutdown_connections 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb3e4240 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbceff5640 0 client.25549 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cf920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cf3e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2466903182 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 >> 172.21.15.23:0/2466903182 conn(0x55b4fb3c3030 msgr2=0x55b4fb3c4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 shutdown_connections 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2466903182 wait complete. 2024-03-28T15:54:52.703 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.688+0000 7ffbf3bff740 20 client.25549 trim_cache size 0 max 16384 2024-03-28T15:54:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.702+0000 7ffbf3bff740 1 Processor -- start 2024-03-28T15:54:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.702+0000 7ffbf3bff740 1 -- start start 2024-03-28T15:54:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.702+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.704 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.702+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.702+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1ae980 con 0x55b4fb3d1e00 2024-03-28T15:54:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1b0300 con 0x55b4fb3c9030 2024-03-28T15:54:52.705 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1aefe0 con 0x55b4fb3c8710 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbeb59e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:33302/0 (socket says 172.21.15.23:33302) 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 -- 172.21.15.23:0/746044731 learned_addr learned my addr 172.21.15.23:0/746044731 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:58826/0 (socket says 172.21.15.23:58826) 2024-03-28T15:54:52.707 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 -- 172.21.15.23:0/746044731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 msgr2=0x55b4fb3cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 -- 172.21.15.23:0/746044731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 msgr2=0x55b4fb3cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 -- 172.21.15.23:0/746044731 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4fb3e45b0 con 0x55b4fb3c8710 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbead9d640 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0b9b80 tx=0x7ffbdc0dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e713a10bd7192efc server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbd0ff9640 1 -- 172.21.15.23:0/746044731 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc04b040 con 0x55b4fb3c8710 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbd0ff9640 1 -- 172.21.15.23:0/746044731 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbdc02ccd0 con 0x55b4fb3c8710 2024-03-28T15:54:52.708 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.703+0000 7ffbd0ff9640 1 -- 172.21.15.23:0/746044731 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbdc0bd1b0 con 0x55b4fb3c8710 2024-03-28T15:54:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 -- 172.21.15.23:0/746044731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 msgr2=0x55b4fb3d15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc0b9b80 tx=0x7ffbdc0dd040 comp rx=0 tx=0).stop 2024-03-28T15:54:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 -- 172.21.15.23:0/746044731 shutdown_connections 2024-03-28T15:54:52.712 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/746044731 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d15d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 -- 172.21.15.23:0/746044731 >> 172.21.15.23:0/746044731 conn(0x55b4fb3c3030 msgr2=0x55b4fb3c3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 -- 172.21.15.23:0/746044731 shutdown_connections 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.704+0000 7ffbf3bff740 1 -- 172.21.15.23:0/746044731 wait complete. 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 -- start start 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 Processor -- start 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1afca0 con 0x55b4fb3c8710 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1af640 con 0x55b4fb3c9030 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1acb40 con 0x55b4fb3d1e00 2024-03-28T15:54:52.713 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbf3bff740 10 client.0 _ll_register_callbacks cb 0x7ffbe0009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbead9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbeb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:58836/0 (socket says 172.21.15.23:58836) 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 -- 172.21.15.23:0/2302845345 learned_addr learned my addr 172.21.15.23:0/2302845345 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbeb59e640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:33310/0 (socket says 172.21.15.23:33310) 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 -- 172.21.15.23:0/2302845345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 msgr2=0x55b4fb3d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 -- 172.21.15.23:0/2302845345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 msgr2=0x55b4fb3d9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbea59c640 1 -- 172.21.15.23:0/2302845345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b4fb3d5480 con 0x55b4fb3c9030 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.705+0000 7ffbeb59e640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbea59c640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0000c00 tx=0x7ffbe0009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=424e463b3891c8aa server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:54:52.714 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe001ca80 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ffbe001cc20 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b4fb2b2710 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbe0005070 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 10 client.25613 fetch_fsmap learned FSMap version 29 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55b4fb43d4a0 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b4fb2b2710 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ffbe0004a70 con 0x55b4fb3c9030 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 10 client.25613 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 10 client.25613 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 20 client.25613 populate_metadata read hostname 'smithi023' 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb43fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 4 client.25613 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:54:52.715 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.706+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b4fb2fe630 con 0x55b4fb3e1e50 2024-03-28T15:54:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.707+0000 7ffbead9d640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb43fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:54:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.707+0000 7ffbceff5640 10 client.25613 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.707+0000 7ffbead9d640 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb43fc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd371337a0c4a997 server_cookie=7263ce6ea37df505 in_seq=0 out_seq=0 2024-03-28T15:54:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.717+0000 7ffbceff5640 1 -- 172.21.15.23:0/2302845345 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b4fb2fe630 con 0x55b4fb3e1e50 2024-03-28T15:54:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.717+0000 7ffbceff5640 10 client.25613 handle_command_reply: tid=0 2024-03-28T15:54:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.717+0000 7ffbf3bff740 1 client.25613 shutdown 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 msgr2=0x55b4fb3d4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ffbe0000c00 tx=0x7ffbe0009640 comp rx=0 tx=0).stop 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 shutdown_connections 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55b4fb3e1e50 0x55b4fb43fc30 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55b4fb3d1e00 0x55b4fb3d4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b4fb3c9030 0x55b4fb3d4a00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbceff5640 0 client.25613 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 --2- 172.21.15.23:0/2302845345 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55b4fb3c8710 0x55b4fb3d9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 >> 172.21.15.23:0/2302845345 conn(0x55b4fb3c3030 msgr2=0x55b4fb3cd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 shutdown_connections 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 1 -- 172.21.15.23:0/2302845345 wait complete. 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf3bff740 20 client.25613 trim_cache size 0 max 16384 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:54:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:54:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 msgr2=0x7ffbd40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 0x7ffbd40a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffbe4009fd0 tx=0x7ffbe40116b0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 msgr2=0x7ffbec197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:54:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.718+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc005940 tx=0x7ffbdc004270 comp rx=0 tx=0).stop 2024-03-28T15:54:52.727 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 shutdown_connections 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ffbd40a2a10 0x7ffbd40a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec106660 0x7ffbec19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ffbec100f60 0x7ffbec198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 --2- 172.21.15.23:0/591377936 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ffbec1005a0 0x7ffbec197e30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 >> 172.21.15.23:0/591377936 conn(0x7ffbec077700 msgr2=0x7ffbec0fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 shutdown_connections 2024-03-28T15:54:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:54:52.719+0000 7ffbf1012640 1 -- 172.21.15.23:0/591377936 wait complete. 2024-03-28T15:54:54.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:54 smithi117 ceph-mon[36002]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:54.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:54 smithi120 ceph-mon[36826]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:54 smithi023 ceph-mon[32448]: pgmap v57: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:56 smithi117 ceph-mon[36002]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:56 smithi120 ceph-mon[36826]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:56 smithi023 ceph-mon[32448]: pgmap v58: 129 pgs: 129 active+clean; 2.5 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 861 KiB/s wr, 10 op/s 2024-03-28T15:54:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:54:58 smithi117 ceph-mon[36002]: pgmap v59: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:54:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:54:58 smithi120 ceph-mon[36826]: pgmap v59: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:54:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:54:58 smithi023 ceph-mon[32448]: pgmap v59: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:55:00.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:00 smithi120 ceph-mon[36826]: pgmap v60: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 863 KiB/s wr, 14 op/s 2024-03-28T15:55:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:00 smithi023 ceph-mon[32448]: pgmap v60: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 863 KiB/s wr, 14 op/s 2024-03-28T15:55:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:00 smithi117 ceph-mon[36002]: pgmap v60: 129 pgs: 129 active+clean; 2.6 MiB data, 363 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 863 KiB/s wr, 14 op/s 2024-03-28T15:55:02.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:02 smithi120 ceph-mon[36826]: pgmap v61: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-03-28T15:55:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:02 smithi023 ceph-mon[32448]: pgmap v61: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-03-28T15:55:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:02 smithi117 ceph-mon[36002]: pgmap v61: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.3 MiB/s wr, 42 op/s 2024-03-28T15:55:04.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:04 smithi120 ceph-mon[36826]: pgmap v62: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:04 smithi023 ceph-mon[32448]: pgmap v62: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:04 smithi117 ceph-mon[36002]: pgmap v62: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:05.839 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 -- 172.21.15.23:0/1986993400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181005a0 msgr2=0x7ff918100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/1986993400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181005a0 0x7ff918100a20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff900009890 tx=0x7ff90002bb70 comp rx=0 tx=0).stop 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 -- 172.21.15.23:0/1986993400 shutdown_connections 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/1986993400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918100f60 0x7ff91810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/1986993400 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff9181005a0 0x7ff918100a20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/1986993400 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff918106660 0x7ff918106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 -- 172.21.15.23:0/1986993400 >> 172.21.15.23:0/1986993400 conn(0x7ff918077700 msgr2=0x7ff918077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 -- 172.21.15.23:0/1986993400 shutdown_connections 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.839+0000 7ff91d3f4640 1 -- 172.21.15.23:0/1986993400 wait complete. 2024-03-28T15:55:05.840 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 Processor -- start 2024-03-28T15:55:05.842 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 -- start start 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 0x7ff91819a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 0x7ff91819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 0x7ff91819e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91810f8a0 con 0x7ff9181005a0 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91810f720 con 0x7ff918106660 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff91d3f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91810fa20 con 0x7ff918100f60 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 0x7ff91819a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff90e7fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 0x7ff91819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 0x7ff91819a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39496/0 (socket says 172.21.15.23:39496) 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff916ffd640 1 -- 172.21.15.23:0/2017751089 learned_addr learned my addr 172.21.15.23:0/2017751089 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.840+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 0x7ff91819e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.843 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 -- 172.21.15.23:0/2017751089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 msgr2=0x7ff91819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 0x7ff91819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 -- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 msgr2=0x7ff91819a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 0x7ff91819a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 -- 172.21.15.23:0/2017751089 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91819f0f0 con 0x7ff918106660 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 0x7ff91819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 0x7ff91819e9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff910009fd0 tx=0x7ff91000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff914ff9640 1 -- 172.21.15.23:0/2017751089 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91000ce20 con 0x7ff918106660 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff914ff9640 1 -- 172.21.15.23:0/2017751089 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff910005020 con 0x7ff918106660 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.841+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91819f3e0 con 0x7ff918106660 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.842+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9181a6c70 con 0x7ff918106660 2024-03-28T15:55:05.844 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.842+0000 7ff914ff9640 1 -- 172.21.15.23:0/2017751089 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff910010720 con 0x7ff918106660 2024-03-28T15:55:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.844+0000 7ff914ff9640 1 -- 172.21.15.23:0/2017751089 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff910005540 con 0x7ff918106660 2024-03-28T15:55:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 Processor -- start 2024-03-28T15:55:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 -- start start 2024-03-28T15:55:05.852 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061d6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061e21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x56250613ab10 con 0x5625061d7030 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625060e3900 con 0x5625061d6710 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x562506136bb0 con 0x5625061dfe00 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff914ff9640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 0x7ff9040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39498/0 (socket says 172.21.15.23:39498) 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 -- 172.21.15.23:0/3648972148 learned_addr learned my addr 172.21.15.23:0/3648972148 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:05.853 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061d6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061e21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 0x7ff9040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 -- 172.21.15.23:0/3648972148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 msgr2=0x5625061e21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061e21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 -- 172.21.15.23:0/3648972148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 msgr2=0x5625061d6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff916ffd640 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061d6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff916ffd640 1 -- 172.21.15.23:0/3648972148 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562505e4f210 con 0x5625061d7030 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.845+0000 7ff914ff9640 1 -- 172.21.15.23:0/2017751089 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ff9100cf860 con 0x7ff918106660 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff916ffd640 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff908004600 tx=0x7ff90800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd271743ae4cdf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.854 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff9177fe640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 0x7ff9040a4ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9100cd1e0 tx=0x7ff9100c6540 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff90d7fa640 1 -- 172.21.15.23:0/3648972148 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90801c070 con 0x5625061d7030 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff90d7fa640 1 -- 172.21.15.23:0/3648972148 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff908009460 con 0x5625061d7030 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.846+0000 7ff90d7fa640 1 -- 172.21.15.23:0/3648972148 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908017630 con 0x5625061d7030 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- 172.21.15.23:0/3648972148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 msgr2=0x5625061df8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7ff908004600 tx=0x7ff90800dac0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- 172.21.15.23:0/3648972148 shutdown_connections 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061e21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d7030 0x5625061df8c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/3648972148 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061d6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- 172.21.15.23:0/3648972148 >> 172.21.15.23:0/3648972148 conn(0x5625061d1030 msgr2=0x5625061d1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- 172.21.15.23:0/3648972148 shutdown_connections 2024-03-28T15:55:05.855 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- 172.21.15.23:0/3648972148 wait complete. 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 -- start start 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.847+0000 7ff91ffdd740 1 Processor -- start 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061dd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061dd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625061e2a30 con 0x5625061dfe00 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625060f6d50 con 0x5625061d6710 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625061e2bb0 con 0x5625061d7030 2024-03-28T15:55:05.856 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff91ffdd740 10 client.0 _ll_register_callbacks cb 0x5625060f6720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34728/0 (socket says 172.21.15.23:34728) 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 -- 172.21.15.23:0/2654639063 learned_addr learned my addr 172.21.15.23:0/2654639063 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061dd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061dd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 -- 172.21.15.23:0/2654639063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 msgr2=0x5625061dd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061dd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 -- 172.21.15.23:0/2654639063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 msgr2=0x5625061dd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061dd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 -- 172.21.15.23:0/2654639063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562505f1b620 con 0x5625061d6710 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061dd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-28T15:55:05.857 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff900009890 tx=0x7ff90002bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafa4da7421a47f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.848+0000 7ff8fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff900039040 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff908009460 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ff900005460 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff900056050 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 10 client.25625 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5625061eec90 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff908009460 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff90003f040 con 0x5625061d6710 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 10 client.25625 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 10 client.25625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 20 client.25625 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:05.858 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625061f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 4 client.25625 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.849+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562506148050 con 0x5625061f0100 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.850+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625061f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.850+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625061f24f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cab4e36228a840be server_cookie=d44f828c8ecd0c61 in_seq=0 out_seq=0 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.850+0000 7ff8fe7fc640 10 client.25625 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/2654639063 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562506148050 con 0x5625061f0100 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff8fe7fc640 10 client.25625 handle_command_reply: tid=0 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 client.25625 shutdown 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 msgr2=0x5625061dcdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff900009890 tx=0x7ff90002bb70 comp rx=0 tx=0).stop 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 shutdown_connections 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625061f24f0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff8fe7fc640 0 client.25625 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:05.859 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061dd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061dd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2654639063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061dcdf0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 >> 172.21.15.23:0/2654639063 conn(0x5625061d1030 msgr2=0x5625061d2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 shutdown_connections 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2654639063 wait complete. 2024-03-28T15:55:05.860 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.851+0000 7ff91ffdd740 20 client.25625 trim_cache size 0 max 16384 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 Processor -- start 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 -- start start 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 0x5625061df520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061eb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181b0ce0 con 0x5625061d6710 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181b2660 con 0x5625061d7030 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.870+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181b1340 con 0x5625061dfe00 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff90e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 0x5625061df520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.871 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:34738/0 (socket says 172.21.15.23:34738) 2024-03-28T15:55:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff90e7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 0x5625061df520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39510/0 (socket says 172.21.15.23:39510) 2024-03-28T15:55:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 -- 172.21.15.23:0/2524487894 learned_addr learned my addr 172.21.15.23:0/2524487894 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:05.872 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061eb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 -- 172.21.15.23:0/2524487894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 msgr2=0x5625061eb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061eb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 -- 172.21.15.23:0/2524487894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 msgr2=0x5625061df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 0x5625061df520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 -- 172.21.15.23:0/2524487894 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5625061d8990 con 0x5625061d7030 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff916ffd640 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff908004600 tx=0x7ff908000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bc3cb26d78f650f server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.871+0000 7ff90cff9640 1 -- 172.21.15.23:0/2524487894 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90801c040 con 0x5625061d7030 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.872+0000 7ff90cff9640 1 -- 172.21.15.23:0/2524487894 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff908028070 con 0x5625061d7030 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.872+0000 7ff90cff9640 1 -- 172.21.15.23:0/2524487894 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9080048e0 con 0x5625061d7030 2024-03-28T15:55:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.872+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2524487894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 msgr2=0x5625061eb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.872+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff908004600 tx=0x7ff908000bc0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2524487894 shutdown_connections 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061dfe00 0x5625061eb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d7030 0x5625061eb2c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/2524487894 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061d6710 0x5625061df520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2524487894 >> 172.21.15.23:0/2524487894 conn(0x5625061d1030 msgr2=0x5625061d7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2524487894 shutdown_connections 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 -- 172.21.15.23:0/2524487894 wait complete. 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 -- start start 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 Processor -- start 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061e33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.873+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff91ffdd740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181b2000 con 0x5625061dfe00 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181b19a0 con 0x5625061d6710 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff91ffdd740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9181aeea0 con 0x5625061d7030 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff91ffdd740 10 client.0 _ll_register_callbacks cb 0x7ff900034580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:48178/0 (socket says 172.21.15.23:48178) 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 -- 172.21.15.23:0/4176547276 learned_addr learned my addr 172.21.15.23:0/4176547276 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061e33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 -- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 msgr2=0x5625061e33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061e33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 -- 172.21.15.23:0/4176547276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 msgr2=0x5625061e3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061e3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 -- 172.21.15.23:0/4176547276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5625062531b0 con 0x5625061d7030 2024-03-28T15:55:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff916ffd640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff908009af0 tx=0x7ff9080094f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5089a93a3964f1f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.874+0000 7ff8fe7fc640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90801c040 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff908028070 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7ff908029270 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff90803e050 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 10 client.25573 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5625062544c0 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff908028070 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7ff908016070 con 0x5625061d7030 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 10 client.25573 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 10 client.25573 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 20 client.25573 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625062569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 4 client.25573 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:55:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.875+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5625061423a0 con 0x5625061f0100 2024-03-28T15:55:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.876+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625062569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.876+0000 7ff90e7fc640 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625062569a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc930474bdcafe09 server_cookie=cb1d2b9f23c459cb in_seq=0 out_seq=0 2024-03-28T15:55:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.876+0000 7ff8fe7fc640 10 client.25573 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff8fe7fc640 1 -- 172.21.15.23:0/4176547276 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5625061423a0 con 0x5625061f0100 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff8fe7fc640 10 client.25573 handle_command_reply: tid=0 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 client.25573 shutdown 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 msgr2=0x5625061e3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff908009af0 tx=0x7ff9080094f0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 shutdown_connections 2024-03-28T15:55:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x5625061f0100 0x5625062569a0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x5625061dfe00 0x5625061e3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff8fe7fc640 0 client.25573 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x5625061d7030 0x5625061e3900 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 --2- 172.21.15.23:0/4176547276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5625061d6710 0x5625061e33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 >> 172.21.15.23:0/4176547276 conn(0x5625061d1030 msgr2=0x5625061dcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 shutdown_connections 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 1 -- 172.21.15.23:0/4176547276 wait complete. 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.886+0000 7ff91ffdd740 20 client.25573 trim_cache size 0 max 16384 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:55:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:55:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:55:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:55:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 msgr2=0x7ff9040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 0x7ff9040a4ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9100cd1e0 tx=0x7ff9100c6540 comp rx=0 tx=0).stop 2024-03-28T15:55:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 msgr2=0x7ff91819e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 0x7ff91819e9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff910009fd0 tx=0x7ff91000ef10 comp rx=0 tx=0).stop 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 shutdown_connections 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff9040a2a10 0x7ff9040a4ed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff918106660 0x7ff91819e9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff918100f60 0x7ff91819a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 --2- 172.21.15.23:0/2017751089 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff9181005a0 0x7ff91819a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 >> 172.21.15.23:0/2017751089 conn(0x7ff918077700 msgr2=0x7ff9180ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 shutdown_connections 2024-03-28T15:55:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:05.887+0000 7ff91d3f4640 1 -- 172.21.15.23:0/2017751089 wait complete. 2024-03-28T15:55:06.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:06 smithi120 ceph-mon[36826]: pgmap v63: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:06 smithi023 ceph-mon[32448]: pgmap v63: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:06 smithi117 ceph-mon[36002]: pgmap v63: 129 pgs: 129 active+clean; 106 MiB data, 366 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 919 KiB/s wr, 37 op/s 2024-03-28T15:55:08.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:08 smithi120 ceph-mon[36826]: pgmap v64: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 302 op/s 2024-03-28T15:55:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:08 smithi023 ceph-mon[32448]: pgmap v64: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 302 op/s 2024-03-28T15:55:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:08 smithi117 ceph-mon[36002]: pgmap v64: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 MiB/s wr, 302 op/s 2024-03-28T15:55:10.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:10 smithi120 ceph-mon[36826]: pgmap v65: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 MiB/s wr, 294 op/s 2024-03-28T15:55:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:10 smithi023 ceph-mon[32448]: pgmap v65: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 MiB/s wr, 294 op/s 2024-03-28T15:55:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:10 smithi117 ceph-mon[36002]: pgmap v65: 129 pgs: 129 active+clean; 256 MiB data, 475 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 MiB/s wr, 294 op/s 2024-03-28T15:55:12.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:12 smithi120 ceph-mon[36826]: pgmap v66: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 500 op/s 2024-03-28T15:55:12.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:12 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:12 smithi023 ceph-mon[32448]: pgmap v66: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 500 op/s 2024-03-28T15:55:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:12 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:12 smithi117 ceph-mon[36002]: pgmap v66: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 13 MiB/s wr, 500 op/s 2024-03-28T15:55:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:12 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:55:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:13 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:13 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:13.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:13 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:55:13.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:13 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:13.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:13 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:13.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:13 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:55:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:13 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:55:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:13 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:55:13.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:13 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:55:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:14 smithi117 ceph-mon[36002]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:14 smithi120 ceph-mon[36826]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:14 smithi023 ceph-mon[32448]: pgmap v67: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:16 smithi117 ceph-mon[36002]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:16 smithi120 ceph-mon[36826]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:16.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:16 smithi023 ceph-mon[32448]: pgmap v68: 129 pgs: 129 active+clean; 259 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 MiB/s wr, 472 op/s 2024-03-28T15:55:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:18 smithi117 ceph-mon[36002]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 658 op/s 2024-03-28T15:55:18.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:18 smithi120 ceph-mon[36826]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 658 op/s 2024-03-28T15:55:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:18 smithi023 ceph-mon[32448]: pgmap v69: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 14 MiB/s wr, 658 op/s 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 -- 172.21.15.23:0/3622010126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894074360 msgr2=0x7f3894112110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 --2- 172.21.15.23:0/3622010126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894074360 0x7f3894112110 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3884009860 tx=0x7f388402bc30 comp rx=0 tx=0).stop 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 -- 172.21.15.23:0/3622010126 shutdown_connections 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 --2- 172.21.15.23:0/3622010126 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894112650 0x7f3894114a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 --2- 172.21.15.23:0/3622010126 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894074360 0x7f3894112110 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 --2- 172.21.15.23:0/3622010126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894073a40 0x7f3894073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.039 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.038+0000 7f389c463640 1 -- 172.21.15.23:0/3622010126 >> 172.21.15.23:0/3622010126 conn(0x7f38940fd130 msgr2=0x7f38940ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.040 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.040+0000 7f389c463640 1 -- 172.21.15.23:0/3622010126 shutdown_connections 2024-03-28T15:55:19.041 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.040+0000 7f389c463640 1 -- 172.21.15.23:0/3622010126 wait complete. 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.041+0000 7f389c463640 1 Processor -- start 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.041+0000 7f389c463640 1 -- start start 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894073a40 0x7f389419e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894074360 0x7f389419ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941180a0 con 0x7f3894112650 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894117f20 con 0x7f3894074360 2024-03-28T15:55:19.044 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389c463640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894118220 con 0x7f3894073a40 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40412/0 (socket says 172.21.15.23:40412) 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 -- 172.21.15.23:0/3429495510 learned_addr learned my addr 172.21.15.23:0/3429495510 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389a1ff640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894073a40 0x7f389419e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f38999fe640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894074360 0x7f389419ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 -- 172.21.15.23:0/3429495510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894073a40 msgr2=0x7f389419e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894073a40 0x7f389419e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 -- 172.21.15.23:0/3429495510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894074360 msgr2=0x7f389419ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894074360 0x7f389419ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.045 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.042+0000 7f389aa00640 1 -- 172.21.15.23:0/3429495510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38941a34b0 con 0x7f3894112650 2024-03-28T15:55:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.043+0000 7f389aa00640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f388c009fa0 tx=0x7f388c00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.044+0000 7f388b7fe640 1 -- 172.21.15.23:0/3429495510 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c004b20 con 0x7f3894112650 2024-03-28T15:55:19.046 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.044+0000 7f388b7fe640 1 -- 172.21.15.23:0/3429495510 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c003e50 con 0x7f3894112650 2024-03-28T15:55:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.044+0000 7f388b7fe640 1 -- 172.21.15.23:0/3429495510 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00b570 con 0x7f3894112650 2024-03-28T15:55:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.044+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38941a37a0 con 0x7f3894112650 2024-03-28T15:55:19.047 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.045+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38941058e0 con 0x7f3894112650 2024-03-28T15:55:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.049+0000 7f388b7fe640 1 -- 172.21.15.23:0/3429495510 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f388c01d020 con 0x7f3894112650 2024-03-28T15:55:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.050+0000 7f388b7fe640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 0x7f38800a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.050+0000 7f388b7fe640 1 -- 172.21.15.23:0/3429495510 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f388c0cf140 con 0x7f3894112650 2024-03-28T15:55:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389f04a740 1 Processor -- start 2024-03-28T15:55:19.051 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389a1ff640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 0x7f38800a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389f04a740 1 -- start start 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389f04a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428772af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389f04a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428773030 0x55a42877b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.051+0000 7f389f04a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 0x55a42877e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389f04a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4286d6b10 con 0x55a428773030 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389f04a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42867f900 con 0x55a42877be00 2024-03-28T15:55:19.052 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389f04a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4286d2bb0 con 0x55a428772710 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389a1ff640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 0x7f38800a50c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3890005d00 tx=0x7f389000b410 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389a1ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428772af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389a1ff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428772af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:47534/0 (socket says 172.21.15.23:47534) 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389a1ff640 1 -- 172.21.15.23:0/2620613456 learned_addr learned my addr 172.21.15.23:0/2620613456 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f38999fe640 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428773030 0x55a42877b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.053 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.052+0000 7f389aa00640 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 0x55a42877e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 -- 172.21.15.23:0/2620613456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 msgr2=0x55a428772af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428772af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 -- 172.21.15.23:0/2620613456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428773030 msgr2=0x55a42877b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428773030 0x55a42877b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 -- 172.21.15.23:0/2620613456 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4283eb210 con 0x55a42877be00 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f389aa00640 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 0x55a42877e1f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cc460 tx=0x7f388c004280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=724c8ce6363c29f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.053+0000 7f3888bfe640 1 -- 172.21.15.23:0/2620613456 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c01a040 con 0x55a42877be00 2024-03-28T15:55:19.054 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.054+0000 7f3888bfe640 1 -- 172.21.15.23:0/2620613456 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c004cc0 con 0x55a42877be00 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 -- 172.21.15.23:0/2620613456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 msgr2=0x55a42877e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 0x55a42877e1f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cc460 tx=0x7f388c004280 comp rx=0 tx=0).stop 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 -- 172.21.15.23:0/2620613456 shutdown_connections 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a42877be00 0x55a42877e1f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428773030 0x55a42877b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 --2- 172.21.15.23:0/2620613456 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428772af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.055 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.055+0000 7f389f04a740 1 -- 172.21.15.23:0/2620613456 >> 172.21.15.23:0/2620613456 conn(0x55a42876d030 msgr2=0x55a42876d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.056 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.056+0000 7f389f04a740 1 -- 172.21.15.23:0/2620613456 shutdown_connections 2024-03-28T15:55:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.056+0000 7f389f04a740 1 -- 172.21.15.23:0/2620613456 wait complete. 2024-03-28T15:55:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 -- start start 2024-03-28T15:55:19.057 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 Processor -- start 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428779320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a428779860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42877ea30 con 0x55a428772710 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a428692d50 con 0x55a428773030 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42877ebb0 con 0x55a42877be00 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389f04a740 10 client.0 _ll_register_callbacks cb 0x55a428692720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.057+0000 7f389a1ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.058 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40424/0 (socket says 172.21.15.23:40424) 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 -- 172.21.15.23:0/1031398337 learned_addr learned my addr 172.21.15.23:0/1031398337 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f38999fe640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428779320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389aa00640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a428779860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 -- 172.21.15.23:0/1031398337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 msgr2=0x55a428779860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a428779860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 -- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 msgr2=0x55a428779320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.059 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428779320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.058+0000 7f389a1ff640 1 -- 172.21.15.23:0/1031398337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4284b7620 con 0x55a428772710 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.059+0000 7f389a1ff640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3890011890 tx=0x7f3890004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0536eb569c84725 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.059+0000 7f387dffb640 10 client.0 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f389001ee10 con 0x55a428772710 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f3890028470 con 0x55a428772710 2024-03-28T15:55:19.060 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f388c004cc0 con 0x55a428772710 2024-03-28T15:55:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3890028670 con 0x55a428772710 2024-03-28T15:55:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f389f04a740 10 client.15804 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.060+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a42878ad80 con 0x55a428772710 2024-03-28T15:55:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mon.0 v2:172.21.15.23:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c004cc0 con 0x55a428772710 2024-03-28T15:55:19.061 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mon.0 v2:172.21.15.23:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f389001fa80 con 0x55a428772710 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 10 client.15804 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 10 client.15804 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 20 client.15804 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a42878c240 0x55a42878e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 4 client.15804 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a4286e4050 con 0x55a42878c240 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.061+0000 7f38999fe640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a42878c240 0x55a42878e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.062+0000 7f38999fe640 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a42878c240 0x55a42878e630 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb2462b5970b2c4c server_cookie=d0c184b55edca396 in_seq=0 out_seq=0 2024-03-28T15:55:19.062 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.062+0000 7f387dffb640 10 client.15804 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.064+0000 7f387dffb640 1 -- 172.21.15.23:0/1031398337 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a4286e4050 con 0x55a42878c240 2024-03-28T15:55:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.064+0000 7f387dffb640 10 client.15804 handle_command_reply: tid=0 2024-03-28T15:55:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.064+0000 7f389f04a740 1 client.15804 shutdown 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 msgr2=0x55a428778de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3890011890 tx=0x7f3890004730 comp rx=0 tx=0).stop 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 shutdown_connections 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a42878c240 0x55a42878e630 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a428779860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428779320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 --2- 172.21.15.23:0/1031398337 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428778de0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 >> 172.21.15.23:0/1031398337 conn(0x55a42876d030 msgr2=0x55a42876ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.065 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f387dffb640 0 client.15804 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.065+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 shutdown_connections 2024-03-28T15:55:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.066+0000 7f389f04a740 1 -- 172.21.15.23:0/1031398337 wait complete. 2024-03-28T15:55:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.067+0000 7f389f04a740 20 client.15804 trim_cache size 0 max 16384 2024-03-28T15:55:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 Processor -- start 2024-03-28T15:55:19.096 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 -- start start 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428787310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428787850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941bd8b0 con 0x55a42877be00 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941bf230 con 0x55a428773030 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.096+0000 7f389f04a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941bdf10 con 0x55a428772710 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.098 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:40440/0 (socket says 172.21.15.23:40440) 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 -- 172.21.15.23:0/3164296745 learned_addr learned my addr 172.21.15.23:0/3164296745 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f38999fe640 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428787850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389a1ff640 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428787310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 -- 172.21.15.23:0/3164296745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 msgr2=0x55a428787310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428787310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 -- 172.21.15.23:0/3164296745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 msgr2=0x55a428787850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428787850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.099 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.097+0000 7f389aa00640 1 -- 172.21.15.23:0/3164296745 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a428771ba0 con 0x55a42877be00 2024-03-28T15:55:19.100 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.098+0000 7f389aa00640 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f388c099620 tx=0x7f388c004100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7a6fb78c918e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.098+0000 7f387ffff640 1 -- 172.21.15.23:0/3164296745 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c01a070 con 0x55a42877be00 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.098+0000 7f387ffff640 1 -- 172.21.15.23:0/3164296745 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c09f4c0 con 0x55a42877be00 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.098+0000 7f387ffff640 1 -- 172.21.15.23:0/3164296745 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c09f120 con 0x55a42877be00 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 -- 172.21.15.23:0/3164296745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 msgr2=0x55a42877a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f388c099620 tx=0x7f388c004100 comp rx=0 tx=0).stop 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 -- 172.21.15.23:0/3164296745 shutdown_connections 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a42877be00 0x55a42877a500 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428787850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 --2- 172.21.15.23:0/3164296745 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a428772710 0x55a428787310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.099+0000 7f389f04a740 1 -- 172.21.15.23:0/3164296745 >> 172.21.15.23:0/3164296745 conn(0x55a42876d030 msgr2=0x55a4287740e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.100+0000 7f389f04a740 1 -- 172.21.15.23:0/3164296745 shutdown_connections 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.100+0000 7f389f04a740 1 -- 172.21.15.23:0/3164296745 wait complete. 2024-03-28T15:55:19.101 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.100+0000 7f389f04a740 1 -- start start 2024-03-28T15:55:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.100+0000 7f389f04a740 1 Processor -- start 2024-03-28T15:55:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428775ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428775ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.102 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a4287e6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941bebd0 con 0x55a428772710 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941be570 con 0x55a428773030 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389410dae0 con 0x55a42877be00 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389f04a740 10 client.0 _ll_register_callbacks cb 0x7f3890011510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f38999fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428775ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389aa00640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a4287e6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f38999fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428775ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:37160/0 (socket says 172.21.15.23:37160) 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f38999fe640 1 -- 172.21.15.23:0/2131995926 learned_addr learned my addr 172.21.15.23:0/2131995926 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.101+0000 7f389a1ff640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428775ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 -- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 msgr2=0x55a428775ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428775ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 -- 172.21.15.23:0/2131995926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 msgr2=0x55a428775ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:19.103 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428775ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 -- 172.21.15.23:0/2131995926 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a42877e1f0 con 0x55a42877be00 2024-03-28T15:55:19.104 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.102+0000 7f389aa00640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a4287e6a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f388c0bb4a0 tx=0x7f388c09b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91bad9aebd69da9d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.103+0000 7f387dffb640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.103+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c0a7030 con 0x55a42877be00 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.104+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f388c09b5d0 con 0x55a42877be00 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.104+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a4286de3a0 con 0x55a42877be00 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.105+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f388c01a070 con 0x55a42877be00 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.105+0000 7f389f04a740 10 client.25591 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:19.105 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.105+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a4287e8010 con 0x55a42877be00 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a4286de3a0 con 0x55a42877be00 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f388c00b040 con 0x55a42877be00 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 10 client.25591 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 10 client.25591 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 20 client.25591 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:19.107 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a4287e8240 0x55a4287ea630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:19.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 4 client.25591 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:55:19.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a42865c710 con 0x55a4287e8240 2024-03-28T15:55:19.108 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.107+0000 7f389a1ff640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a4287e8240 0x55a4287ea630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.111+0000 7f387dffb640 10 client.25591 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:19.112 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.111+0000 7f389a1ff640 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a4287e8240 0x55a4287ea630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7975bae65aadbef6 server_cookie=68450231f413eba7 in_seq=0 out_seq=0 2024-03-28T15:55:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f387dffb640 1 -- 172.21.15.23:0/2131995926 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a42865c710 con 0x55a4287e8240 2024-03-28T15:55:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f387dffb640 10 client.25591 handle_command_reply: tid=0 2024-03-28T15:55:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 client.25591 shutdown 2024-03-28T15:55:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 msgr2=0x55a4287e6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.246 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a4287e6a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f388c0bb4a0 tx=0x7f388c09b4f0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 shutdown_connections 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a4287e8240 0x55a4287ea630 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a42877be00 0x55a4287e6a90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f387dffb640 0 client.25591 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a428773030 0x55a428775ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 --2- 172.21.15.23:0/2131995926 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a428772710 0x55a428775ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 >> 172.21.15.23:0/2131995926 conn(0x55a42876d030 msgr2=0x55a428778da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 shutdown_connections 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.245+0000 7f389f04a740 1 -- 172.21.15.23:0/2131995926 wait complete. 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.246+0000 7f389f04a740 20 client.25591 trim_cache size 0 max 16384 2024-03-28T15:55:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:55:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:55:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:55:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 msgr2=0x7f38800a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 0x7f38800a50c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3890005d00 tx=0x7f389000b410 comp rx=0 tx=0).stop 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 msgr2=0x7f38941a2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f388c009fa0 tx=0x7f388c00c800 comp rx=0 tx=0).stop 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 shutdown_connections 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f38800a2c00 0x7f38800a50c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3894112650 0x7f38941a2db0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3894074360 0x7f389419ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 --2- 172.21.15.23:0/3429495510 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3894073a40 0x7f389419e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 >> 172.21.15.23:0/3429495510 conn(0x7f38940fd130 msgr2=0x7f38940fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 shutdown_connections 2024-03-28T15:55:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:19.247+0000 7f389c463640 1 -- 172.21.15.23:0/3429495510 wait complete. 2024-03-28T15:55:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:20 smithi117 ceph-mon[36002]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 392 op/s 2024-03-28T15:55:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:20 smithi120 ceph-mon[36826]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 392 op/s 2024-03-28T15:55:20.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:20 smithi023 ceph-mon[32448]: pgmap v70: 129 pgs: 129 active+clean; 259 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 2.7 MiB/s wr, 392 op/s 2024-03-28T15:55:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:22 smithi117 ceph-mon[36002]: pgmap v71: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 MiB/s wr, 1.75k op/s 2024-03-28T15:55:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:22 smithi120 ceph-mon[36826]: pgmap v71: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 MiB/s wr, 1.75k op/s 2024-03-28T15:55:22.922 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:22 smithi023 ceph-mon[32448]: pgmap v71: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 MiB/s wr, 1.75k op/s 2024-03-28T15:55:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:24 smithi117 ceph-mon[36002]: pgmap v72: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:24 smithi120 ceph-mon[36826]: pgmap v72: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:24 smithi023 ceph-mon[32448]: pgmap v72: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:26 smithi117 ceph-mon[36002]: pgmap v73: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:26 smithi120 ceph-mon[36826]: pgmap v73: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:26.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:26 smithi023 ceph-mon[32448]: pgmap v73: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 MiB/s wr, 1.54k op/s 2024-03-28T15:55:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:28 smithi117 ceph-mon[36002]: pgmap v74: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-03-28T15:55:28.842 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:28 smithi120 ceph-mon[36826]: pgmap v74: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-03-28T15:55:28.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:28 smithi023 ceph-mon[32448]: pgmap v74: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 15 MiB/s wr, 1.55k op/s 2024-03-28T15:55:30.722 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:30 smithi023 ceph-mon[32448]: pgmap v75: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-03-28T15:55:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:30 smithi117 ceph-mon[36002]: pgmap v75: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-03-28T15:55:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:30 smithi120 ceph-mon[36826]: pgmap v75: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 MiB/s wr, 1.36k op/s 2024-03-28T15:55:32.353 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 -- 172.21.15.23:0/4064208135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330101640 msgr2=0x7f8330101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 --2- 172.21.15.23:0/4064208135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330101640 0x7f8330101ac0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f831c009930 tx=0x7f831c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 -- 172.21.15.23:0/4064208135 shutdown_connections 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 --2- 172.21.15.23:0/4064208135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330102000 0x7f833010e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 --2- 172.21.15.23:0/4064208135 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330101640 0x7f8330101ac0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 --2- 172.21.15.23:0/4064208135 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330107650 0x7f8330107a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 -- 172.21.15.23:0/4064208135 >> 172.21.15.23:0/4064208135 conn(0x7f83300fd130 msgr2=0x7f83300ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 -- 172.21.15.23:0/4064208135 shutdown_connections 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.353+0000 7f8337174640 1 -- 172.21.15.23:0/4064208135 wait complete. 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 Processor -- start 2024-03-28T15:55:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 -- start start 2024-03-28T15:55:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 0x7f83300739a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102000 0x7f8330073ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330111a60 con 0x7f8330107650 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301118e0 con 0x7f8330101640 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8337174640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330111be0 con 0x7f8330102000 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60250/0 (socket says 172.21.15.23:60250) 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 -- 172.21.15.23:0/2726246652 learned_addr learned my addr 172.21.15.23:0/2726246652 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f832ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102000 0x7f8330073ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.358 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8334f10640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 0x7f83300739a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 -- 172.21.15.23:0/2726246652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102000 msgr2=0x7f8330073ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102000 0x7f8330073ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 -- 172.21.15.23:0/2726246652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 msgr2=0x7f83300739a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 0x7f83300739a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8335711640 1 -- 172.21.15.23:0/2726246652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83301a63c0 con 0x7f8330107650 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.354+0000 7f8334f10640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 0x7f83300739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f8335711640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f832800eb10 tx=0x7f832800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f832dffb640 1 -- 172.21.15.23:0/2726246652 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328004b20 con 0x7f8330107650 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f832dffb640 1 -- 172.21.15.23:0/2726246652 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8328004cc0 con 0x7f8330107650 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83301a66b0 con 0x7f8330107650 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f832dffb640 1 -- 172.21.15.23:0/2726246652 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8328010760 con 0x7f8330107650 2024-03-28T15:55:32.359 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.355+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83301a6b80 con 0x7f8330107650 2024-03-28T15:55:32.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f832dffb640 1 -- 172.21.15.23:0/2726246652 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8328005020 con 0x7f8330107650 2024-03-28T15:55:32.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f8339e37740 1 Processor -- start 2024-03-28T15:55:32.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f8339e37740 1 -- start start 2024-03-28T15:55:32.362 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f832dffb640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 0x7f83200a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f832dffb640 1 -- 172.21.15.23:0/2726246652 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f83280cf770 con 0x7f8330107650 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f8339e37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.358+0000 7f8339e37740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff458c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8339e37740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff481f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8339e37740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82fea0b10 con 0x55a82ff45e00 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8339e37740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82fe49900 con 0x55a82ff3d030 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8339e37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82fe9cbb0 con 0x55a82ff3c710 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f832ffff640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 0x7f83200a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:40488/0 (socket says 172.21.15.23:40488) 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 -- 172.21.15.23:0/3459311934 learned_addr learned my addr 172.21.15.23:0/3459311934 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f832ffff640 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff458c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8335711640 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff481f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 -- 172.21.15.23:0/3459311934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 msgr2=0x55a82ff458c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff458c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 -- 172.21.15.23:0/3459311934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 msgr2=0x55a82ff481f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff481f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 -- 172.21.15.23:0/3459311934 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a82fbb5210 con 0x55a82ff3c710 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f832ffff640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 0x7f83200a4f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f831c005ab0 tx=0x7f831c004e70 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f8334f10640 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83240114f0 tx=0x7f83240118f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b370884f744adabb server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f831affd640 1 -- 172.21.15.23:0/3459311934 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832401e070 con 0x55a82ff3c710 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f831affd640 1 -- 172.21.15.23:0/3459311934 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f832400f9c0 con 0x55a82ff3c710 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.359+0000 7f831affd640 1 -- 172.21.15.23:0/3459311934 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324019a60 con 0x55a82ff3c710 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.360+0000 7f8339e37740 1 -- 172.21.15.23:0/3459311934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 msgr2=0x55a82ff3caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.360+0000 7f8339e37740 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f83240114f0 tx=0x7f83240118f0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 -- 172.21.15.23:0/3459311934 shutdown_connections 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff481f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff458c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 --2- 172.21.15.23:0/3459311934 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff3caf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 -- 172.21.15.23:0/3459311934 >> 172.21.15.23:0/3459311934 conn(0x55a82ff37030 msgr2=0x55a82ff37410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 -- 172.21.15.23:0/3459311934 shutdown_connections 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 -- 172.21.15.23:0/3459311934 wait complete. 2024-03-28T15:55:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 -- start start 2024-03-28T15:55:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.362+0000 7f8339e37740 1 Processor -- start 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff43840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f832ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60272/0 (socket says 172.21.15.23:60272) 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f832ffff640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:40500/0 (socket says 172.21.15.23:40500) 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82ff48a30 con 0x55a82ff3c710 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 -- 172.21.15.23:0/4095552602 learned_addr learned my addr 172.21.15.23:0/4095552602 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82fe5cd50 con 0x55a82ff45e00 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a82ff48bb0 con 0x55a82ff3d030 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8335711640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff43840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 10 client.0 _ll_register_callbacks cb 0x55a82fe5c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 -- 172.21.15.23:0/4095552602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 msgr2=0x55a82ff43300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 -- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 msgr2=0x55a82ff43840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff43840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8334f10640 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a82fc81620 con 0x55a82ff3c710 2024-03-28T15:55:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f832ffff640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8335711640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff43840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.363+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f832400f9c0 con 0x55a82ff3c710 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.364+0000 7f8334f10640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f832400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48b0dc1003bd1dc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.364+0000 7f830ffff640 10 client.15822 ms_handle_connect on v2:172.21.15.23:3300/0 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.364+0000 7f830ffff640 1 -- 172.21.15.23:0/4095552602 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832401e040 con 0x55a82ff3c710 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.364+0000 7f830ffff640 1 -- 172.21.15.23:0/4095552602 <== mon.0 v2:172.21.15.23:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f8324005050 con 0x55a82ff3c710 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.364+0000 7f830ffff640 1 -- 172.21.15.23:0/4095552602 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8324005290 con 0x55a82ff3c710 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 10 client.15822 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:32.371 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({fsmap=29}) v3 -- 0x55a82ff54b50 con 0x55a82ff3c710 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f830ffff640 1 -- 172.21.15.23:0/4095552602 <== mon.0 v2:172.21.15.23:3300/0 4 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f832402a030 con 0x55a82ff3c710 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 10 client.15822 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 20 client.15822 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ff55ea0 0x55a82ff58290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 4 client.15822 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a82feae050 con 0x55a82ff55ea0 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.368+0000 7f832ffff640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ff55ea0 0x55a82ff58290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.372 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.369+0000 7f830ffff640 10 client.15822 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:32.373 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.369+0000 7f832ffff640 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ff55ea0 0x55a82ff58290 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1dd136c0adbab72 server_cookie=8c613e705711f98f in_seq=0 out_seq=0 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f830ffff640 1 -- 172.21.15.23:0/4095552602 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a82feae050 con 0x55a82ff55ea0 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f830ffff640 10 client.15822 handle_command_reply: tid=0 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 client.15822 shutdown 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 msgr2=0x55a82ff42dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f832400f710 comp rx=0 tx=0).stop 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 shutdown_connections 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ff55ea0 0x55a82ff58290 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff43840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.374 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f830ffff640 0 client.15822 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3d030 0x55a82ff43300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 --2- 172.21.15.23:0/4095552602 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3c710 0x55a82ff42dc0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 >> 172.21.15.23:0/4095552602 conn(0x55a82ff37030 msgr2=0x55a82ff38d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 shutdown_connections 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 1 -- 172.21.15.23:0/4095552602 wait complete. 2024-03-28T15:55:32.375 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.370+0000 7f8339e37740 20 client.15822 trim_cache size 0 max 16384 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 Processor -- start 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 -- start start 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff454f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301b0c80 con 0x55a82ff45e00 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301b2600 con 0x55a82ff3d030 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.384+0000 7f8339e37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301b12e0 con 0x55a82ff3c710 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.390 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f8335711640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:36670/0 (socket says 172.21.15.23:36670) 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 -- 172.21.15.23:0/2435462754 learned_addr learned my addr 172.21.15.23:0/2435462754 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f8335711640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:60286/0 (socket says 172.21.15.23:60286) 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f8334f10640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff454f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 -- 172.21.15.23:0/2435462754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 msgr2=0x55a82ff454f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff454f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 -- 172.21.15.23:0/2435462754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 msgr2=0x55a82ff512c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 -- 172.21.15.23:0/2435462754 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a82ff3ce60 con 0x55a82ff3d030 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f8334f10640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f8335711640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f832ffff640 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f831c004ee0 tx=0x7f831c02bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c1199397e88c3d server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.391 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f831a7fc640 1 -- 172.21.15.23:0/2435462754 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0510d0 con 0x55a82ff3d030 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f831a7fc640 1 -- 172.21.15.23:0/2435462754 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f831c057070 con 0x55a82ff3d030 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.385+0000 7f831a7fc640 1 -- 172.21.15.23:0/2435462754 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0051b0 con 0x55a82ff3d030 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- 172.21.15.23:0/2435462754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 msgr2=0x55a82ff43660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f831c004ee0 tx=0x7f831c02bc30 comp rx=0 tx=0).stop 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- 172.21.15.23:0/2435462754 shutdown_connections 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff45e00 0x55a82ff512c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff3d030 0x55a82ff43660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 --2- 172.21.15.23:0/2435462754 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff454f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- 172.21.15.23:0/2435462754 >> 172.21.15.23:0/2435462754 conn(0x55a82ff37030 msgr2=0x55a82ff37410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- 172.21.15.23:0/2435462754 shutdown_connections 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- 172.21.15.23:0/2435462754 wait complete. 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.386+0000 7f8339e37740 1 -- start start 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 Processor -- start 2024-03-28T15:55:32.392 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3d030 0x55a82ff48dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff49300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301b1fa0 con 0x55a82ff3d030 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301b1940 con 0x55a82ff45e00 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83301aee40 con 0x55a82ff3c710 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8339e37740 10 client.0 _ll_register_callbacks cb 0x7f8324009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f832ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3d030 0x55a82ff48dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:40516/0 (socket says 172.21.15.23:40516) 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 -- 172.21.15.23:0/3523882569 learned_addr learned my addr 172.21.15.23:0/3523882569 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8335711640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff49300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 -- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 msgr2=0x55a82ff49300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.393 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff49300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 -- 172.21.15.23:0/3523882569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3d030 msgr2=0x55a82ff48dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-28T15:55:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3d030 0x55a82ff48dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.394 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.387+0000 7f8334f10640 1 -- 172.21.15.23:0/3523882569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a82ff49940 con 0x55a82ff3c710 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8334f10640 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f8324005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e127d5247798c17e server_cookie=0 in_seq=0 out_seq=0 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 10 client.0 ms_handle_connect on v2:172.21.15.120:3300/0 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832401e040 con 0x55a82ff3c710 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mon.1 v2:172.21.15.120:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9495+0+0 (secure 0 0 0) 0x7f83240183b0 con 0x55a82ff3c710 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f831c057070 con 0x55a82ff3c710 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f832403c050 con 0x55a82ff3c710 2024-03-28T15:55:32.395 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 10 client.25609 fetch_fsmap learned FSMap version 29 2024-03-28T15:55:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a82ffb13c0 con 0x55a82ff3c710 2024-03-28T15:55:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mon.1 v2:172.21.15.120:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f831c057070 con 0x55a82ff3c710 2024-03-28T15:55:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mon.1 v2:172.21.15.120:3300/0 5 ==== fsmap(e 29) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f832402a070 con 0x55a82ff3c710 2024-03-28T15:55:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 10 client.25609 fetch_fsmap finished waiting for FSMap version 29 2024-03-28T15:55:32.396 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 10 client.25609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 20 client.25609 populate_metadata read hostname 'smithi023' 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.388+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ffb1a00 0x55a82ffb3e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.389+0000 7f8339e37740 4 client.25609 mds_command: new command op to 24494 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.389+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 --> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a82fea83a0 con 0x55a82ffb1a00 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.389+0000 7f832ffff640 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ffb1a00 0x55a82ffb3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.389+0000 7f830ffff640 10 client.25609 ms_handle_connect on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:32.397 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.389+0000 7f832ffff640 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ffb1a00 0x55a82ffb3e20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e70bdd4b5baf17cf server_cookie=1c04d29091256f4f in_seq=0 out_seq=0 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.399+0000 7f830ffff640 1 -- 172.21.15.23:0/3523882569 <== mds.0 v2:172.21.15.117:6838/3971000510 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a82fea83a0 con 0x55a82ffb1a00 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.399+0000 7f830ffff640 10 client.25609 handle_command_reply: tid=0 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.399+0000 7f8339e37740 1 client.25609 shutdown 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.399+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 msgr2=0x55a82ff4da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.399+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8324000c00 tx=0x7f8324005f90 comp rx=0 tx=0).stop 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 shutdown_connections 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:6838/3971000510,v1:172.21.15.117:6839/3971000510] conn(0x55a82ffb1a00 0x55a82ffb3e20 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a82ff45e00 0x55a82ff49300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x55a82ff3d030 0x55a82ff48dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.401 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 --2- 172.21.15.23:0/3523882569 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x55a82ff3c710 0x55a82ff4da00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f830ffff640 0 client.25609 ms_handle_reset on v2:172.21.15.117:6838/3971000510 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 >> 172.21.15.23:0/3523882569 conn(0x55a82ff37030 msgr2=0x55a82ff35790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 shutdown_connections 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 1 -- 172.21.15.23:0/3523882569 wait complete. 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8339e37740 20 client.25609 trim_cache size 0 max 16384 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:{ 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "message": "", 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout: "return_code": 0 2024-03-28T15:55:32.402 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stdout:} 2024-03-28T15:55:32.403 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 msgr2=0x7f83200a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 0x7f83200a4f00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f831c005ab0 tx=0x7f831c004e70 comp rx=0 tx=0).stop 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 msgr2=0x7f8330074420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f832800eb10 tx=0x7f832800c2d0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 shutdown_connections 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f83200a2a40 0x7f83200a4f00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f8330107650 0x7f8330074420 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f8330102000 0x7f8330073ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.404 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 --2- 172.21.15.23:0/2726246652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8330101640 0x7f83300739a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T15:55:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.400+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 >> 172.21.15.23:0/2726246652 conn(0x7f83300fd130 msgr2=0x7f83300fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T15:55:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.401+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 shutdown_connections 2024-03-28T15:55:32.405 INFO:teuthology.task.background_exec.ubuntu@smithi023.front.sepia.ceph.com.smithi023.stderr:2024-03-28T15:55:32.401+0000 7f8337174640 1 -- 172.21.15.23:0/2726246652 wait complete. 2024-03-28T15:55:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:32 smithi117 ceph-mon[36002]: pgmap v76: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-03-28T15:55:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:32 smithi120 ceph-mon[36826]: pgmap v76: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-03-28T15:55:32.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:32 smithi023 ceph-mon[32448]: pgmap v76: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 14 MiB/s wr, 1.37k op/s 2024-03-28T15:55:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:34 smithi117 ceph-mon[36002]: pgmap v77: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:34 smithi120 ceph-mon[36826]: pgmap v77: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:34 smithi023 ceph-mon[32448]: pgmap v77: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:36.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:36 smithi117 ceph-mon[36002]: pgmap v78: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:36 smithi120 ceph-mon[36826]: pgmap v78: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:36 smithi023 ceph-mon[32448]: pgmap v78: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 866 KiB/s wr, 10 op/s 2024-03-28T15:55:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:38 smithi117 ceph-mon[36002]: pgmap v79: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:55:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:38 smithi120 ceph-mon[36826]: pgmap v79: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:55:38.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:38 smithi023 ceph-mon[32448]: pgmap v79: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 1.3 MiB/s wr, 19 op/s 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:setting up random write file 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:done setting up random write file 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:starting fsync run 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0078s 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0053s 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0048s 2024-03-28T15:55:40.286 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0065s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0067s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0049s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0063s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0055s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0052s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0049s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0066s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0047s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0052s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0065s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.287 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0057s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0052s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0059s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0058s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0047s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0056s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0048s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0054s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0066s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0054s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0057s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0052s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0054s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0053s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0059s 2024-03-28T15:55:40.288 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0060s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0067s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0063s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0053s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0054s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0004s fsync time: 0.0157s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0126s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0054s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0047s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0050s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0049s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0067s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0050s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0048s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0052s 2024-03-28T15:55:40.289 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0055s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0066s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0058s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0060s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0050s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0049s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0051s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0048s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0002s fsync time: 0.0055s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0049s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:write time: 0.0003s fsync time: 0.0055s 2024-03-28T15:55:40.290 INFO:tasks.workunit.client.0.smithi023.stdout:run done 61 fsyncs total, killing random writer 2024-03-28T15:55:40.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ echo /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-28T15:55:40.291 INFO:tasks.workunit.client.0.smithi023.stdout:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-03-28T15:55:40.291 INFO:tasks.workunit.client.0.smithi023.stderr:+ whereis lsof 2024-03-28T15:55:40.324 INFO:tasks.workunit.client.0.smithi023.stdout:lsof: /usr/bin/lsof /usr/share/man/man1/lsof.1.gz 2024-03-28T15:55:40.324 INFO:tasks.workunit.client.0.smithi023.stderr:+ lsof 2024-03-28T15:55:40.348 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/a1effab339b46693621f83d849345b020354fac9c91be6d17236ad20a0b73f09/merged 2024-03-28T15:55:40.348 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/f735f5f96a26be5db08a5931a68c05aed51050f6c123368dafa5ba27a44e125d/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/160d8dc4b8fb4bcf5af9fc4089b7229eb3d34323a38f3c113b04638349a8f79f/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/8ac9f704d3276e021ac8932b9e84d915e21d94d036c118999e87184beaece252/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/29eda3d6a669f7553aaf2f04a876c6cd8ac7eab1139fd8f09451b9b073915090/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/d3a25d8d50b398d6a3010de5ac3c6f2cbfeb96c573a23f8cbef1202a5c5c5b21/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/c60356adfbee0fa15457daabb126aad0d641c7ee29776e1f340c6185b75cbe18/merged 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.349 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/bcb73bbb3fd8dc95cc1280d14fc486873d6e2e956ea44496aec984de587d007f/merged 2024-03-28T15:55:40.350 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.350 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/881a3aab06f4a8c61a81d9c5b0d03b40a5ad0322a9f4a30e41443b15c0e13940/merged 2024-03-28T15:55:40.350 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.350 INFO:tasks.workunit.client.0.smithi023.stderr:lsof: WARNING: can't stat() overlay file system /var/lib/containers/storage/overlay/ec950c9912ec25530a3bad89eb7bf806aba9718918aabbc5b4c30c425f53ba76/merged 2024-03-28T15:55:40.350 INFO:tasks.workunit.client.0.smithi023.stderr: Output information may be incomplete. 2024-03-28T15:55:40.409 INFO:tasks.workunit.client.0.smithi023.stdout:COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 1 root cwd unknown /proc/1/cwd (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 1 root rtd unknown /proc/1/root (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 1 root txt unknown /proc/1/exe (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 1 root NOFD /proc/1/fd (opendir: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:kthreadd 2 root cwd unknown /proc/2/cwd (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:kthreadd 2 root rtd unknown /proc/2/root (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:kthreadd 2 root txt unknown /proc/2/exe (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:kthreadd 2 root NOFD /proc/2/fd (opendir: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_gp 3 root cwd unknown /proc/3/cwd (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_gp 3 root rtd unknown /proc/3/root (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_gp 3 root txt unknown /proc/3/exe (readlink: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_gp 3 root NOFD /proc/3/fd (opendir: Permission denied) 2024-03-28T15:55:40.410 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_par_g 4 root cwd unknown /proc/4/cwd (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_par_g 4 root rtd unknown /proc/4/root (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_par_g 4 root txt unknown /proc/4/exe (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_par_g 4 root NOFD /proc/4/fd (opendir: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:slub_flus 5 root cwd unknown /proc/5/cwd (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:slub_flus 5 root rtd unknown /proc/5/root (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:slub_flus 5 root txt unknown /proc/5/exe (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:slub_flus 5 root NOFD /proc/5/fd (opendir: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:netns 6 root cwd unknown /proc/6/cwd (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:netns 6 root rtd unknown /proc/6/root (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:netns 6 root txt unknown /proc/6/exe (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:netns 6 root NOFD /proc/6/fd (opendir: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 8 root cwd unknown /proc/8/cwd (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 8 root rtd unknown /proc/8/root (readlink: Permission denied) 2024-03-28T15:55:40.411 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 8 root txt unknown /proc/8/exe (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 8 root NOFD /proc/8/fd (opendir: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:mm_percpu 10 root cwd unknown /proc/10/cwd (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:mm_percpu 10 root rtd unknown /proc/10/root (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:mm_percpu 10 root txt unknown /proc/10/exe (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:mm_percpu 10 root NOFD /proc/10/fd (opendir: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 11 root cwd unknown /proc/11/cwd (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 11 root rtd unknown /proc/11/root (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 11 root txt unknown /proc/11/exe (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 11 root NOFD /proc/11/fd (opendir: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 12 root cwd unknown /proc/12/cwd (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 12 root rtd unknown /proc/12/root (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 12 root txt unknown /proc/12/exe (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 12 root NOFD /proc/12/fd (opendir: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 13 root cwd unknown /proc/13/cwd (readlink: Permission denied) 2024-03-28T15:55:40.412 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 13 root rtd unknown /proc/13/root (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 13 root txt unknown /proc/13/exe (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 13 root NOFD /proc/13/fd (opendir: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 14 root cwd unknown /proc/14/cwd (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 14 root rtd unknown /proc/14/root (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 14 root txt unknown /proc/14/exe (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_tasks 14 root NOFD /proc/14/fd (opendir: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 15 root cwd unknown /proc/15/cwd (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 15 root rtd unknown /proc/15/root (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 15 root txt unknown /proc/15/exe (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 15 root NOFD /proc/15/fd (opendir: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:pr/ttyS1 16 root cwd unknown /proc/16/cwd (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:pr/ttyS1 16 root rtd unknown /proc/16/root (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:pr/ttyS1 16 root txt unknown /proc/16/exe (readlink: Permission denied) 2024-03-28T15:55:40.413 INFO:tasks.workunit.client.0.smithi023.stdout:pr/ttyS1 16 root NOFD /proc/16/fd (opendir: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_preem 17 root cwd unknown /proc/17/cwd (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_preem 17 root rtd unknown /proc/17/root (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_preem 17 root txt unknown /proc/17/exe (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:rcu_preem 17 root NOFD /proc/17/fd (opendir: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:migration 18 root cwd unknown /proc/18/cwd (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:migration 18 root rtd unknown /proc/18/root (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:migration 18 root txt unknown /proc/18/exe (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:migration 18 root NOFD /proc/18/fd (opendir: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 19 root cwd unknown /proc/19/cwd (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 19 root rtd unknown /proc/19/root (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 19 root txt unknown /proc/19/exe (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 19 root NOFD /proc/19/fd (opendir: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/0 21 root cwd unknown /proc/21/cwd (readlink: Permission denied) 2024-03-28T15:55:40.414 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/0 21 root rtd unknown /proc/21/root (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/0 21 root txt unknown /proc/21/exe (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/0 21 root NOFD /proc/21/fd (opendir: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/1 22 root cwd unknown /proc/22/cwd (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/1 22 root rtd unknown /proc/22/root (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/1 22 root txt unknown /proc/22/exe (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/1 22 root NOFD /proc/22/fd (opendir: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 23 root cwd unknown /proc/23/cwd (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 23 root rtd unknown /proc/23/root (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 23 root txt unknown /proc/23/exe (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 23 root NOFD /proc/23/fd (opendir: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:migration 24 root cwd unknown /proc/24/cwd (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:migration 24 root rtd unknown /proc/24/root (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:migration 24 root txt unknown /proc/24/exe (readlink: Permission denied) 2024-03-28T15:55:40.415 INFO:tasks.workunit.client.0.smithi023.stdout:migration 24 root NOFD /proc/24/fd (opendir: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 25 root cwd unknown /proc/25/cwd (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 25 root rtd unknown /proc/25/root (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 25 root txt unknown /proc/25/exe (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 25 root NOFD /proc/25/fd (opendir: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 27 root cwd unknown /proc/27/cwd (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 27 root rtd unknown /proc/27/root (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 27 root txt unknown /proc/27/exe (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 27 root NOFD /proc/27/fd (opendir: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/2 28 root cwd unknown /proc/28/cwd (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/2 28 root rtd unknown /proc/28/root (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/2 28 root txt unknown /proc/28/exe (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/2 28 root NOFD /proc/28/fd (opendir: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 29 root cwd unknown /proc/29/cwd (readlink: Permission denied) 2024-03-28T15:55:40.416 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 29 root rtd unknown /proc/29/root (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 29 root txt unknown /proc/29/exe (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 29 root NOFD /proc/29/fd (opendir: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:migration 30 root cwd unknown /proc/30/cwd (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:migration 30 root rtd unknown /proc/30/root (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:migration 30 root txt unknown /proc/30/exe (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:migration 30 root NOFD /proc/30/fd (opendir: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 31 root cwd unknown /proc/31/cwd (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 31 root rtd unknown /proc/31/root (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 31 root txt unknown /proc/31/exe (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 31 root NOFD /proc/31/fd (opendir: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 33 root cwd unknown /proc/33/cwd (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 33 root rtd unknown /proc/33/root (readlink: Permission denied) 2024-03-28T15:55:40.417 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 33 root txt unknown /proc/33/exe (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 33 root NOFD /proc/33/fd (opendir: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/3 34 root cwd unknown /proc/34/cwd (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/3 34 root rtd unknown /proc/34/root (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/3 34 root txt unknown /proc/34/exe (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/3 34 root NOFD /proc/34/fd (opendir: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 35 root cwd unknown /proc/35/cwd (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 35 root rtd unknown /proc/35/root (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 35 root txt unknown /proc/35/exe (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 35 root NOFD /proc/35/fd (opendir: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:migration 36 root cwd unknown /proc/36/cwd (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:migration 36 root rtd unknown /proc/36/root (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:migration 36 root txt unknown /proc/36/exe (readlink: Permission denied) 2024-03-28T15:55:40.418 INFO:tasks.workunit.client.0.smithi023.stdout:migration 36 root NOFD /proc/36/fd (opendir: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 37 root cwd unknown /proc/37/cwd (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 37 root rtd unknown /proc/37/root (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 37 root txt unknown /proc/37/exe (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 37 root NOFD /proc/37/fd (opendir: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 39 root cwd unknown /proc/39/cwd (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 39 root rtd unknown /proc/39/root (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 39 root txt unknown /proc/39/exe (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 39 root NOFD /proc/39/fd (opendir: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/4 40 root cwd unknown /proc/40/cwd (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/4 40 root rtd unknown /proc/40/root (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/4 40 root txt unknown /proc/40/exe (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/4 40 root NOFD /proc/40/fd (opendir: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 41 root cwd unknown /proc/41/cwd (readlink: Permission denied) 2024-03-28T15:55:40.419 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 41 root rtd unknown /proc/41/root (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 41 root txt unknown /proc/41/exe (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 41 root NOFD /proc/41/fd (opendir: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:migration 42 root cwd unknown /proc/42/cwd (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:migration 42 root rtd unknown /proc/42/root (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:migration 42 root txt unknown /proc/42/exe (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:migration 42 root NOFD /proc/42/fd (opendir: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 43 root cwd unknown /proc/43/cwd (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 43 root rtd unknown /proc/43/root (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 43 root txt unknown /proc/43/exe (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 43 root NOFD /proc/43/fd (opendir: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 45 root cwd unknown /proc/45/cwd (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 45 root rtd unknown /proc/45/root (readlink: Permission denied) 2024-03-28T15:55:40.420 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 45 root txt unknown /proc/45/exe (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 45 root NOFD /proc/45/fd (opendir: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/5 46 root cwd unknown /proc/46/cwd (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/5 46 root rtd unknown /proc/46/root (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/5 46 root txt unknown /proc/46/exe (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/5 46 root NOFD /proc/46/fd (opendir: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 47 root cwd unknown /proc/47/cwd (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 47 root rtd unknown /proc/47/root (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 47 root txt unknown /proc/47/exe (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 47 root NOFD /proc/47/fd (opendir: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:migration 48 root cwd unknown /proc/48/cwd (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:migration 48 root rtd unknown /proc/48/root (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:migration 48 root txt unknown /proc/48/exe (readlink: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:migration 48 root NOFD /proc/48/fd (opendir: Permission denied) 2024-03-28T15:55:40.421 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 49 root cwd unknown /proc/49/cwd (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 49 root rtd unknown /proc/49/root (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 49 root txt unknown /proc/49/exe (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 49 root NOFD /proc/49/fd (opendir: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 50 root cwd unknown /proc/50/cwd (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 50 root rtd unknown /proc/50/root (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 50 root txt unknown /proc/50/exe (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 50 root NOFD /proc/50/fd (opendir: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 51 root cwd unknown /proc/51/cwd (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 51 root rtd unknown /proc/51/root (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 51 root txt unknown /proc/51/exe (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 51 root NOFD /proc/51/fd (opendir: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/6 52 root cwd unknown /proc/52/cwd (readlink: Permission denied) 2024-03-28T15:55:40.422 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/6 52 root rtd unknown /proc/52/root (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/6 52 root txt unknown /proc/52/exe (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/6 52 root NOFD /proc/52/fd (opendir: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 53 root cwd unknown /proc/53/cwd (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 53 root rtd unknown /proc/53/root (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 53 root txt unknown /proc/53/exe (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 53 root NOFD /proc/53/fd (opendir: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:migration 54 root cwd unknown /proc/54/cwd (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:migration 54 root rtd unknown /proc/54/root (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:migration 54 root txt unknown /proc/54/exe (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:migration 54 root NOFD /proc/54/fd (opendir: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 55 root cwd unknown /proc/55/cwd (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 55 root rtd unknown /proc/55/root (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 55 root txt unknown /proc/55/exe (readlink: Permission denied) 2024-03-28T15:55:40.423 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 55 root NOFD /proc/55/fd (opendir: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57 root cwd unknown /proc/57/cwd (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57 root rtd unknown /proc/57/root (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57 root txt unknown /proc/57/exe (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57 root NOFD /proc/57/fd (opendir: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/7 58 root cwd unknown /proc/58/cwd (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/7 58 root rtd unknown /proc/58/root (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/7 58 root txt unknown /proc/58/exe (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:cpuhp/7 58 root NOFD /proc/58/fd (opendir: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 59 root cwd unknown /proc/59/cwd (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 59 root rtd unknown /proc/59/root (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 59 root txt unknown /proc/59/exe (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:idle_inje 59 root NOFD /proc/59/fd (opendir: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:migration 60 root cwd unknown /proc/60/cwd (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:migration 60 root rtd unknown /proc/60/root (readlink: Permission denied) 2024-03-28T15:55:40.424 INFO:tasks.workunit.client.0.smithi023.stdout:migration 60 root txt unknown /proc/60/exe (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:migration 60 root NOFD /proc/60/fd (opendir: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 61 root cwd unknown /proc/61/cwd (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 61 root rtd unknown /proc/61/root (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 61 root txt unknown /proc/61/exe (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:ksoftirqd 61 root NOFD /proc/61/fd (opendir: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 63 root cwd unknown /proc/63/cwd (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 63 root rtd unknown /proc/63/root (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 63 root txt unknown /proc/63/exe (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 63 root NOFD /proc/63/fd (opendir: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 65 root cwd unknown /proc/65/cwd (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 65 root rtd unknown /proc/65/root (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 65 root txt unknown /proc/65/exe (readlink: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 65 root NOFD /proc/65/fd (opendir: Permission denied) 2024-03-28T15:55:40.425 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 67 root cwd unknown /proc/67/cwd (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 67 root rtd unknown /proc/67/root (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 67 root txt unknown /proc/67/exe (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 67 root NOFD /proc/67/fd (opendir: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 70 root cwd unknown /proc/70/cwd (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 70 root rtd unknown /proc/70/root (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 70 root txt unknown /proc/70/exe (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 70 root NOFD /proc/70/fd (opendir: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kdevtmpfs 71 root cwd unknown /proc/71/cwd (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kdevtmpfs 71 root rtd unknown /proc/71/root (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kdevtmpfs 71 root txt unknown /proc/71/exe (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:kdevtmpfs 71 root NOFD /proc/71/fd (opendir: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:inet_frag 72 root cwd unknown /proc/72/cwd (readlink: Permission denied) 2024-03-28T15:55:40.426 INFO:tasks.workunit.client.0.smithi023.stdout:inet_frag 72 root rtd unknown /proc/72/root (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:inet_frag 72 root txt unknown /proc/72/exe (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:inet_frag 72 root NOFD /proc/72/fd (opendir: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:kauditd 73 root cwd unknown /proc/73/cwd (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:kauditd 73 root rtd unknown /proc/73/root (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:kauditd 73 root txt unknown /proc/73/exe (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:kauditd 73 root NOFD /proc/73/fd (opendir: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:khungtask 74 root cwd unknown /proc/74/cwd (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:khungtask 74 root rtd unknown /proc/74/root (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:khungtask 74 root txt unknown /proc/74/exe (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:khungtask 74 root NOFD /proc/74/fd (opendir: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:oom_reape 75 root cwd unknown /proc/75/cwd (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:oom_reape 75 root rtd unknown /proc/75/root (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:oom_reape 75 root txt unknown /proc/75/exe (readlink: Permission denied) 2024-03-28T15:55:40.427 INFO:tasks.workunit.client.0.smithi023.stdout:oom_reape 75 root NOFD /proc/75/fd (opendir: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:writeback 76 root cwd unknown /proc/76/cwd (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:writeback 76 root rtd unknown /proc/76/root (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:writeback 76 root txt unknown /proc/76/exe (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:writeback 76 root NOFD /proc/76/fd (opendir: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:kcompactd 77 root cwd unknown /proc/77/cwd (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:kcompactd 77 root rtd unknown /proc/77/root (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:kcompactd 77 root txt unknown /proc/77/exe (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:kcompactd 77 root NOFD /proc/77/fd (opendir: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:ksmd 78 root cwd unknown /proc/78/cwd (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:ksmd 78 root rtd unknown /proc/78/root (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:ksmd 78 root txt unknown /proc/78/exe (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:ksmd 78 root NOFD /proc/78/fd (opendir: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:khugepage 80 root cwd unknown /proc/80/cwd (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:khugepage 80 root rtd unknown /proc/80/root (readlink: Permission denied) 2024-03-28T15:55:40.428 INFO:tasks.workunit.client.0.smithi023.stdout:khugepage 80 root txt unknown /proc/80/exe (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:khugepage 80 root NOFD /proc/80/fd (opendir: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:cryptd 81 root cwd unknown /proc/81/cwd (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:cryptd 81 root rtd unknown /proc/81/root (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:cryptd 81 root txt unknown /proc/81/exe (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:cryptd 81 root NOFD /proc/81/fd (opendir: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kintegrit 82 root cwd unknown /proc/82/cwd (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kintegrit 82 root rtd unknown /proc/82/root (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kintegrit 82 root txt unknown /proc/82/exe (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kintegrit 82 root NOFD /proc/82/fd (opendir: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kblockd 83 root cwd unknown /proc/83/cwd (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kblockd 83 root rtd unknown /proc/83/root (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kblockd 83 root txt unknown /proc/83/exe (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:kblockd 83 root NOFD /proc/83/fd (opendir: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:blkcg_pun 84 root cwd unknown /proc/84/cwd (readlink: Permission denied) 2024-03-28T15:55:40.429 INFO:tasks.workunit.client.0.smithi023.stdout:blkcg_pun 84 root rtd unknown /proc/84/root (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:blkcg_pun 84 root txt unknown /proc/84/exe (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:blkcg_pun 84 root NOFD /proc/84/fd (opendir: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:tpm_dev_w 86 root cwd unknown /proc/86/cwd (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:tpm_dev_w 86 root rtd unknown /proc/86/root (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:tpm_dev_w 86 root txt unknown /proc/86/exe (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:tpm_dev_w 86 root NOFD /proc/86/fd (opendir: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md 87 root cwd unknown /proc/87/cwd (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md 87 root rtd unknown /proc/87/root (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md 87 root txt unknown /proc/87/exe (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md 87 root NOFD /proc/87/fd (opendir: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md_bitmap 88 root cwd unknown /proc/88/cwd (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md_bitmap 88 root rtd unknown /proc/88/root (readlink: Permission denied) 2024-03-28T15:55:40.430 INFO:tasks.workunit.client.0.smithi023.stdout:md_bitmap 88 root txt unknown /proc/88/exe (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:md_bitmap 88 root NOFD /proc/88/fd (opendir: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:edac-poll 89 root cwd unknown /proc/89/cwd (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:edac-poll 89 root rtd unknown /proc/89/root (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:edac-poll 89 root txt unknown /proc/89/exe (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:edac-poll 89 root NOFD /proc/89/fd (opendir: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:watchdogd 90 root cwd unknown /proc/90/cwd (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:watchdogd 90 root rtd unknown /proc/90/root (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:watchdogd 90 root txt unknown /proc/90/exe (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:watchdogd 90 root NOFD /proc/90/fd (opendir: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 93 root cwd unknown /proc/93/cwd (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 93 root rtd unknown /proc/93/root (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 93 root txt unknown /proc/93/exe (readlink: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 93 root NOFD /proc/93/fd (opendir: Permission denied) 2024-03-28T15:55:40.431 INFO:tasks.workunit.client.0.smithi023.stdout:kswapd0 94 root cwd unknown /proc/94/cwd (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kswapd0 94 root rtd unknown /proc/94/root (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kswapd0 94 root txt unknown /proc/94/exe (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kswapd0 94 root NOFD /proc/94/fd (opendir: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kthrotld 101 root cwd unknown /proc/101/cwd (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kthrotld 101 root rtd unknown /proc/101/root (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kthrotld 101 root txt unknown /proc/101/exe (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kthrotld 101 root NOFD /proc/101/fd (opendir: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 108 root cwd unknown /proc/108/cwd (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 108 root rtd unknown /proc/108/root (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 108 root txt unknown /proc/108/exe (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 108 root NOFD /proc/108/fd (opendir: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:acpi_ther 109 root cwd unknown /proc/109/cwd (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:acpi_ther 109 root rtd unknown /proc/109/root (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:acpi_ther 109 root txt unknown /proc/109/exe (readlink: Permission denied) 2024-03-28T15:55:40.432 INFO:tasks.workunit.client.0.smithi023.stdout:acpi_ther 109 root NOFD /proc/109/fd (opendir: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kmpath_rd 110 root cwd unknown /proc/110/cwd (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kmpath_rd 110 root rtd unknown /proc/110/root (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kmpath_rd 110 root txt unknown /proc/110/exe (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kmpath_rd 110 root NOFD /proc/110/fd (opendir: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kaluad 111 root cwd unknown /proc/111/cwd (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kaluad 111 root rtd unknown /proc/111/root (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kaluad 111 root txt unknown /proc/111/exe (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:kaluad 111 root NOFD /proc/111/fd (opendir: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:mld 112 root cwd unknown /proc/112/cwd (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:mld 112 root rtd unknown /proc/112/root (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:mld 112 root txt unknown /proc/112/exe (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:mld 112 root NOFD /proc/112/fd (opendir: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:ipv6_addr 113 root cwd unknown /proc/113/cwd (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:ipv6_addr 113 root rtd unknown /proc/113/root (readlink: Permission denied) 2024-03-28T15:55:40.433 INFO:tasks.workunit.client.0.smithi023.stdout:ipv6_addr 113 root txt unknown /proc/113/exe (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:ipv6_addr 113 root NOFD /proc/113/fd (opendir: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kstrp 122 root cwd unknown /proc/122/cwd (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kstrp 122 root rtd unknown /proc/122/root (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kstrp 122 root txt unknown /proc/122/exe (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kstrp 122 root NOFD /proc/122/fd (opendir: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 124 root cwd unknown /proc/124/cwd (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 124 root rtd unknown /proc/124/root (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 124 root txt unknown /proc/124/exe (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 124 root NOFD /proc/124/fd (opendir: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:zswap-shr 129 root cwd unknown /proc/129/cwd (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:zswap-shr 129 root rtd unknown /proc/129/root (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:zswap-shr 129 root txt unknown /proc/129/exe (readlink: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:zswap-shr 129 root NOFD /proc/129/fd (opendir: Permission denied) 2024-03-28T15:55:40.434 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 130 root cwd unknown /proc/130/cwd (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 130 root rtd unknown /proc/130/root (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 130 root txt unknown /proc/130/exe (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 130 root NOFD /proc/130/fd (opendir: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 238 root cwd unknown /proc/238/cwd (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 238 root rtd unknown /proc/238/root (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 238 root txt unknown /proc/238/exe (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 238 root NOFD /proc/238/fd (opendir: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 278 root cwd unknown /proc/278/cwd (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 278 root rtd unknown /proc/278/root (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 278 root txt unknown /proc/278/exe (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 278 root NOFD /proc/278/fd (opendir: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 280 root cwd unknown /proc/280/cwd (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 280 root rtd unknown /proc/280/root (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 280 root txt unknown /proc/280/exe (readlink: Permission denied) 2024-03-28T15:55:40.435 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 280 root NOFD /proc/280/fd (opendir: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 281 root cwd unknown /proc/281/cwd (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 281 root rtd unknown /proc/281/root (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 281 root txt unknown /proc/281/exe (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 281 root NOFD /proc/281/fd (opendir: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 287 root cwd unknown /proc/287/cwd (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 287 root rtd unknown /proc/287/root (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 287 root txt unknown /proc/287/exe (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 287 root NOFD /proc/287/fd (opendir: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 295 root cwd unknown /proc/295/cwd (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 295 root rtd unknown /proc/295/root (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 295 root txt unknown /proc/295/exe (readlink: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 295 root NOFD /proc/295/fd (opendir: Permission denied) 2024-03-28T15:55:40.436 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 301 root cwd unknown /proc/301/cwd (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 301 root rtd unknown /proc/301/root (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 301 root txt unknown /proc/301/exe (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 301 root NOFD /proc/301/fd (opendir: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-wq 432 root cwd unknown /proc/432/cwd (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-wq 432 root rtd unknown /proc/432/root (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-wq 432 root txt unknown /proc/432/exe (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-wq 432 root NOFD /proc/432/fd (opendir: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-rese 433 root cwd unknown /proc/433/cwd (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-rese 433 root rtd unknown /proc/433/root (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-rese 433 root txt unknown /proc/433/exe (readlink: Permission denied) 2024-03-28T15:55:40.437 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-rese 433 root NOFD /proc/433/fd (opendir: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-dele 434 root cwd unknown /proc/434/cwd (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-dele 434 root rtd unknown /proc/434/root (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-dele 434 root txt unknown /proc/434/exe (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-dele 434 root NOFD /proc/434/fd (opendir: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-auth 435 root cwd unknown /proc/435/cwd (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-auth 435 root rtd unknown /proc/435/root (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-auth 435 root txt unknown /proc/435/exe (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:nvme-auth 435 root NOFD /proc/435/fd (opendir: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:ixgbe 442 root cwd unknown /proc/442/cwd (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:ixgbe 442 root rtd unknown /proc/442/root (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:ixgbe 442 root txt unknown /proc/442/exe (readlink: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:ixgbe 442 root NOFD /proc/442/fd (opendir: Permission denied) 2024-03-28T15:55:40.438 INFO:tasks.workunit.client.0.smithi023.stdout:ata_sff 443 root cwd unknown /proc/443/cwd (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:ata_sff 443 root rtd unknown /proc/443/root (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:ata_sff 443 root txt unknown /proc/443/exe (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:ata_sff 443 root NOFD /proc/443/fd (opendir: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_0 452 root cwd unknown /proc/452/cwd (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_0 452 root rtd unknown /proc/452/root (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_0 452 root txt unknown /proc/452/exe (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_0 452 root NOFD /proc/452/fd (opendir: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 453 root cwd unknown /proc/453/cwd (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 453 root rtd unknown /proc/453/root (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 453 root txt unknown /proc/453/exe (readlink: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 453 root NOFD /proc/453/fd (opendir: Permission denied) 2024-03-28T15:55:40.439 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_1 454 root cwd unknown /proc/454/cwd (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_1 454 root rtd unknown /proc/454/root (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_1 454 root txt unknown /proc/454/exe (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_1 454 root NOFD /proc/454/fd (opendir: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 455 root cwd unknown /proc/455/cwd (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 455 root rtd unknown /proc/455/root (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 455 root txt unknown /proc/455/exe (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 455 root NOFD /proc/455/fd (opendir: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_2 456 root cwd unknown /proc/456/cwd (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_2 456 root rtd unknown /proc/456/root (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_2 456 root txt unknown /proc/456/exe (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_2 456 root NOFD /proc/456/fd (opendir: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 457 root cwd unknown /proc/457/cwd (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 457 root rtd unknown /proc/457/root (readlink: Permission denied) 2024-03-28T15:55:40.440 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 457 root txt unknown /proc/457/exe (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 457 root NOFD /proc/457/fd (opendir: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_3 458 root cwd unknown /proc/458/cwd (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_3 458 root rtd unknown /proc/458/root (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_3 458 root txt unknown /proc/458/exe (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_3 458 root NOFD /proc/458/fd (opendir: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 459 root cwd unknown /proc/459/cwd (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 459 root rtd unknown /proc/459/root (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 459 root txt unknown /proc/459/exe (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 459 root NOFD /proc/459/fd (opendir: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_4 461 root cwd unknown /proc/461/cwd (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_4 461 root rtd unknown /proc/461/root (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_4 461 root txt unknown /proc/461/exe (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_4 461 root NOFD /proc/461/fd (opendir: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 462 root cwd unknown /proc/462/cwd (readlink: Permission denied) 2024-03-28T15:55:40.441 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 462 root rtd unknown /proc/462/root (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 462 root txt unknown /proc/462/exe (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 462 root NOFD /proc/462/fd (opendir: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_5 463 root cwd unknown /proc/463/cwd (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_5 463 root rtd unknown /proc/463/root (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_5 463 root txt unknown /proc/463/exe (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_5 463 root NOFD /proc/463/fd (opendir: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 464 root cwd unknown /proc/464/cwd (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 464 root rtd unknown /proc/464/root (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 464 root txt unknown /proc/464/exe (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 464 root NOFD /proc/464/fd (opendir: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_6 465 root cwd unknown /proc/465/cwd (readlink: Permission denied) 2024-03-28T15:55:40.442 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_6 465 root rtd unknown /proc/465/root (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_6 465 root txt unknown /proc/465/exe (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_6 465 root NOFD /proc/465/fd (opendir: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 466 root cwd unknown /proc/466/cwd (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 466 root rtd unknown /proc/466/root (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 466 root txt unknown /proc/466/exe (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 466 root NOFD /proc/466/fd (opendir: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_7 467 root cwd unknown /proc/467/cwd (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_7 467 root rtd unknown /proc/467/root (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_7 467 root txt unknown /proc/467/exe (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_7 467 root NOFD /proc/467/fd (opendir: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 468 root cwd unknown /proc/468/cwd (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 468 root rtd unknown /proc/468/root (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 468 root txt unknown /proc/468/exe (readlink: Permission denied) 2024-03-28T15:55:40.443 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 468 root NOFD /proc/468/fd (opendir: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_8 469 root cwd unknown /proc/469/cwd (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_8 469 root rtd unknown /proc/469/root (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_8 469 root txt unknown /proc/469/exe (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_8 469 root NOFD /proc/469/fd (opendir: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 470 root cwd unknown /proc/470/cwd (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 470 root rtd unknown /proc/470/root (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 470 root txt unknown /proc/470/exe (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 470 root NOFD /proc/470/fd (opendir: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_9 471 root cwd unknown /proc/471/cwd (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_9 471 root rtd unknown /proc/471/root (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_9 471 root txt unknown /proc/471/exe (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_eh_9 471 root NOFD /proc/471/fd (opendir: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 472 root cwd unknown /proc/472/cwd (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 472 root rtd unknown /proc/472/root (readlink: Permission denied) 2024-03-28T15:55:40.444 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 472 root txt unknown /proc/472/exe (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:scsi_tmf_ 472 root NOFD /proc/472/fd (opendir: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 474 root cwd unknown /proc/474/cwd (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 474 root rtd unknown /proc/474/root (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 474 root txt unknown /proc/474/exe (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 474 root NOFD /proc/474/fd (opendir: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:jbd2/sda1 500 root cwd unknown /proc/500/cwd (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:jbd2/sda1 500 root rtd unknown /proc/500/root (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:jbd2/sda1 500 root txt unknown /proc/500/exe (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:jbd2/sda1 500 root NOFD /proc/500/fd (opendir: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:ext4-rsv- 501 root cwd unknown /proc/501/cwd (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:ext4-rsv- 501 root rtd unknown /proc/501/root (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:ext4-rsv- 501 root txt unknown /proc/501/exe (readlink: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:ext4-rsv- 501 root NOFD /proc/501/fd (opendir: Permission denied) 2024-03-28T15:55:40.445 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-j 577 root cwd unknown /proc/577/cwd (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-j 577 root rtd unknown /proc/577/root (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-j 577 root txt unknown /proc/577/exe (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-j 577 root NOFD /proc/577/fd (opendir: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:ext4lazyi 583 root cwd unknown /proc/583/cwd (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:ext4lazyi 583 root rtd unknown /proc/583/root (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:ext4lazyi 583 root txt unknown /proc/583/exe (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:ext4lazyi 583 root NOFD /proc/583/fd (opendir: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-u 595 root cwd unknown /proc/595/cwd (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-u 595 root rtd unknown /proc/595/root (readlink: Permission denied) 2024-03-28T15:55:40.446 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-u 595 root txt unknown /proc/595/exe (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-u 595 root NOFD /proc/595/fd (opendir: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:ipmi-msgh 644 root cwd unknown /proc/644/cwd (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:ipmi-msgh 644 root rtd unknown /proc/644/root (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:ipmi-msgh 644 root txt unknown /proc/644/exe (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:ipmi-msgh 644 root NOFD /proc/644/fd (opendir: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:kipmi0 654 root cwd unknown /proc/654/cwd (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:kipmi0 654 root rtd unknown /proc/654/root (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:kipmi0 654 root txt unknown /proc/654/exe (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:kipmi0 654 root NOFD /proc/654/fd (opendir: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:rpcbind 669 rpc cwd unknown /proc/669/cwd (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:rpcbind 669 rpc rtd unknown /proc/669/root (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:rpcbind 669 rpc txt unknown /proc/669/exe (readlink: Permission denied) 2024-03-28T15:55:40.447 INFO:tasks.workunit.client.0.smithi023.stdout:rpcbind 669 rpc NOFD /proc/669/fd (opendir: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:rpciod 672 root cwd unknown /proc/672/cwd (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:rpciod 672 root rtd unknown /proc/672/root (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:rpciod 672 root txt unknown /proc/672/exe (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:rpciod 672 root NOFD /proc/672/fd (opendir: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:xprtiod 673 root cwd unknown /proc/673/cwd (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:xprtiod 673 root rtd unknown /proc/673/root (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:xprtiod 673 root txt unknown /proc/673/exe (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:xprtiod 673 root NOFD /proc/673/fd (opendir: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 root cwd unknown /proc/674/cwd (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 root rtd unknown /proc/674/root (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 root txt unknown /proc/674/exe (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 root NOFD /proc/674/fd (opendir: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 675 auditd root cwd unknown /proc/674/task/675/cwd (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 675 auditd root rtd unknown /proc/674/task/675/root (readlink: Permission denied) 2024-03-28T15:55:40.448 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 675 auditd root txt unknown /proc/674/task/675/exe (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 675 auditd root NOFD /proc/674/task/675/fd (opendir: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 677 auditd root cwd unknown /proc/674/task/677/cwd (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 677 auditd root rtd unknown /proc/674/task/677/root (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 677 auditd root txt unknown /proc/674/task/677/exe (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:auditd 674 677 auditd root NOFD /proc/674/task/677/fd (opendir: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:sedispatc 676 root cwd unknown /proc/676/cwd (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:sedispatc 676 root rtd unknown /proc/676/root (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:sedispatc 676 root txt unknown /proc/676/exe (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:sedispatc 676 root NOFD /proc/676/fd (opendir: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 715 dbus cwd unknown /proc/715/cwd (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 715 dbus rtd unknown /proc/715/root (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 715 dbus txt unknown /proc/715/exe (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 715 dbus NOFD /proc/715/fd (opendir: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 716 dbus cwd unknown /proc/716/cwd (readlink: Permission denied) 2024-03-28T15:55:40.449 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 716 dbus rtd unknown /proc/716/root (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 716 dbus txt unknown /proc/716/exe (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:dbus-brok 716 dbus NOFD /proc/716/fd (opendir: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 root cwd unknown /proc/717/cwd (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 root rtd unknown /proc/717/root (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 root txt unknown /proc/717/exe (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 root NOFD /proc/717/fd (opendir: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 745 gmain root cwd unknown /proc/717/task/745/cwd (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 745 gmain root rtd unknown /proc/717/task/745/root (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 745 gmain root txt unknown /proc/717/task/745/exe (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 745 gmain root NOFD /proc/717/task/745/fd (opendir: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 746 gdbus root cwd unknown /proc/717/task/746/cwd (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 746 gdbus root rtd unknown /proc/717/task/746/root (readlink: Permission denied) 2024-03-28T15:55:40.450 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 746 gdbus root txt unknown /proc/717/task/746/exe (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:NetworkMa 717 746 gdbus root NOFD /proc/717/task/746/fd (opendir: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 root cwd unknown /proc/720/cwd (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 root rtd unknown /proc/720/root (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 root txt unknown /proc/720/exe (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 root NOFD /proc/720/fd (opendir: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 727 gmain root cwd unknown /proc/720/task/727/cwd (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 727 gmain root rtd unknown /proc/720/task/727/root (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 727 gmain root txt unknown /proc/720/task/727/exe (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:irqbalanc 720 727 gmain root NOFD /proc/720/task/727/fd (opendir: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:lsmd 721 libstoragemgmt cwd unknown /proc/721/cwd (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:lsmd 721 libstoragemgmt rtd unknown /proc/721/root (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:lsmd 721 libstoragemgmt txt unknown /proc/721/exe (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:lsmd 721 libstoragemgmt NOFD /proc/721/fd (opendir: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:mcelog 722 root cwd unknown /proc/722/cwd (readlink: Permission denied) 2024-03-28T15:55:40.451 INFO:tasks.workunit.client.0.smithi023.stdout:mcelog 722 root rtd unknown /proc/722/root (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:mcelog 722 root txt unknown /proc/722/exe (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:mcelog 722 root NOFD /proc/722/fd (opendir: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 polkitd cwd unknown /proc/723/cwd (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 polkitd rtd unknown /proc/723/root (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 polkitd txt unknown /proc/723/exe (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 polkitd NOFD /proc/723/fd (opendir: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 735 gmain polkitd cwd unknown /proc/723/task/735/cwd (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 735 gmain polkitd rtd unknown /proc/723/task/735/root (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 735 gmain polkitd txt unknown /proc/723/task/735/exe (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 735 gmain polkitd NOFD /proc/723/task/735/fd (opendir: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 736 gdbus polkitd cwd unknown /proc/723/task/736/cwd (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 736 gdbus polkitd rtd unknown /proc/723/task/736/root (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 736 gdbus polkitd txt unknown /proc/723/task/736/exe (readlink: Permission denied) 2024-03-28T15:55:40.452 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 736 gdbus polkitd NOFD /proc/723/task/736/fd (opendir: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 737 JS\x20Hel polkitd cwd unknown /proc/723/task/737/cwd (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 737 JS\x20Hel polkitd rtd unknown /proc/723/task/737/root (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 737 JS\x20Hel polkitd txt unknown /proc/723/task/737/exe (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 737 JS\x20Hel polkitd NOFD /proc/723/task/737/fd (opendir: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 738 JS\x20Hel polkitd cwd unknown /proc/723/task/738/cwd (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 738 JS\x20Hel polkitd rtd unknown /proc/723/task/738/root (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 738 JS\x20Hel polkitd txt unknown /proc/723/task/738/exe (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 738 JS\x20Hel polkitd NOFD /proc/723/task/738/fd (opendir: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 739 JS\x20Hel polkitd cwd unknown /proc/723/task/739/cwd (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 739 JS\x20Hel polkitd rtd unknown /proc/723/task/739/root (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 739 JS\x20Hel polkitd txt unknown /proc/723/task/739/exe (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 739 JS\x20Hel polkitd NOFD /proc/723/task/739/fd (opendir: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 740 JS\x20Hel polkitd cwd unknown /proc/723/task/740/cwd (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 740 JS\x20Hel polkitd rtd unknown /proc/723/task/740/root (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 740 JS\x20Hel polkitd txt unknown /proc/723/task/740/exe (readlink: Permission denied) 2024-03-28T15:55:40.453 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 740 JS\x20Hel polkitd NOFD /proc/723/task/740/fd (opendir: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 741 JS\x20Hel polkitd cwd unknown /proc/723/task/741/cwd (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 741 JS\x20Hel polkitd rtd unknown /proc/723/task/741/root (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 741 JS\x20Hel polkitd txt unknown /proc/723/task/741/exe (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 741 JS\x20Hel polkitd NOFD /proc/723/task/741/fd (opendir: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 742 JS\x20Hel polkitd cwd unknown /proc/723/task/742/cwd (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 742 JS\x20Hel polkitd rtd unknown /proc/723/task/742/root (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 742 JS\x20Hel polkitd txt unknown /proc/723/task/742/exe (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 742 JS\x20Hel polkitd NOFD /proc/723/task/742/fd (opendir: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 743 JS\x20Hel polkitd cwd unknown /proc/723/task/743/cwd (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 743 JS\x20Hel polkitd rtd unknown /proc/723/task/743/root (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 743 JS\x20Hel polkitd txt unknown /proc/723/task/743/exe (readlink: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 743 JS\x20Hel polkitd NOFD /proc/723/task/743/fd (opendir: Permission denied) 2024-03-28T15:55:40.454 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 744 JS\x20Hel polkitd cwd unknown /proc/723/task/744/cwd (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 744 JS\x20Hel polkitd rtd unknown /proc/723/task/744/root (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 744 JS\x20Hel polkitd txt unknown /proc/723/task/744/exe (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 744 JS\x20Hel polkitd NOFD /proc/723/task/744/fd (opendir: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 749 polkitd polkitd cwd unknown /proc/723/task/749/cwd (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 749 polkitd polkitd rtd unknown /proc/723/task/749/root (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 749 polkitd polkitd txt unknown /proc/723/task/749/exe (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:polkitd 723 749 polkitd polkitd NOFD /proc/723/task/749/fd (opendir: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-l 726 root cwd unknown /proc/726/cwd (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-l 726 root rtd unknown /proc/726/root (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-l 726 root txt unknown /proc/726/exe (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:systemd-l 726 root NOFD /proc/726/fd (opendir: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 779 root cwd unknown /proc/779/cwd (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 779 root rtd unknown /proc/779/root (readlink: Permission denied) 2024-03-28T15:55:40.455 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 779 root txt unknown /proc/779/exe (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 779 root NOFD /proc/779/fd (opendir: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 root cwd unknown /proc/780/cwd (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 root rtd unknown /proc/780/root (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 root txt unknown /proc/780/exe (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 root NOFD /proc/780/fd (opendir: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 865 tuned root cwd unknown /proc/780/task/865/cwd (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 865 tuned root rtd unknown /proc/780/task/865/root (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 865 tuned root txt unknown /proc/780/task/865/exe (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:tuned 780 865 tuned root NOFD /proc/780/task/865/fd (opendir: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 root cwd unknown /proc/784/cwd (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 root rtd unknown /proc/784/root (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 root txt unknown /proc/784/exe (readlink: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 root NOFD /proc/784/fd (opendir: Permission denied) 2024-03-28T15:55:40.456 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 785 gssproxy root cwd unknown /proc/784/task/785/cwd (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 785 gssproxy root rtd unknown /proc/784/task/785/root (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 785 gssproxy root txt unknown /proc/784/task/785/exe (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 785 gssproxy root NOFD /proc/784/task/785/fd (opendir: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 786 gssproxy root cwd unknown /proc/784/task/786/cwd (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 786 gssproxy root rtd unknown /proc/784/task/786/root (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 786 gssproxy root txt unknown /proc/784/task/786/exe (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 786 gssproxy root NOFD /proc/784/task/786/fd (opendir: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 787 gssproxy root cwd unknown /proc/784/task/787/cwd (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 787 gssproxy root rtd unknown /proc/784/task/787/root (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 787 gssproxy root txt unknown /proc/784/task/787/exe (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 787 gssproxy root NOFD /proc/784/task/787/fd (opendir: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 788 gssproxy root cwd unknown /proc/784/task/788/cwd (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 788 gssproxy root rtd unknown /proc/784/task/788/root (readlink: Permission denied) 2024-03-28T15:55:40.457 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 788 gssproxy root txt unknown /proc/784/task/788/exe (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 788 gssproxy root NOFD /proc/784/task/788/fd (opendir: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 789 gssproxy root cwd unknown /proc/784/task/789/cwd (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 789 gssproxy root rtd unknown /proc/784/task/789/root (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 789 gssproxy root txt unknown /proc/784/task/789/exe (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:gssproxy 784 789 gssproxy root NOFD /proc/784/task/789/fd (opendir: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:atd 791 root cwd unknown /proc/791/cwd (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:atd 791 root rtd unknown /proc/791/root (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:atd 791 root txt unknown /proc/791/exe (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:atd 791 root NOFD /proc/791/fd (opendir: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:crond 792 root cwd unknown /proc/792/cwd (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:crond 792 root rtd unknown /proc/792/root (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:crond 792 root txt unknown /proc/792/exe (readlink: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:crond 792 root NOFD /proc/792/fd (opendir: Permission denied) 2024-03-28T15:55:40.458 INFO:tasks.workunit.client.0.smithi023.stdout:nrpe 886 nrpe cwd unknown /proc/886/cwd (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:nrpe 886 nrpe rtd unknown /proc/886/root (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:nrpe 886 nrpe txt unknown /proc/886/exe (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:nrpe 886 nrpe NOFD /proc/886/fd (opendir: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1316 root cwd unknown /proc/1316/cwd (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1316 root rtd unknown /proc/1316/root (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1316 root txt unknown /proc/1316/exe (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1316 root NOFD /proc/1316/fd (opendir: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1321 root cwd unknown /proc/1321/cwd (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1321 root rtd unknown /proc/1321/root (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1321 root txt unknown /proc/1321/exe (readlink: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:agetty 1321 root NOFD /proc/1321/fd (opendir: Permission denied) 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu cwd DIR 8,1 4096 2 / 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu rtd DIR 8,1 4096 2 / 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu txt REG 8,1 102120 20353 /usr/lib/systemd/systemd 2024-03-28T15:55:40.459 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 1167 /etc/selinux/targeted/contexts/files/file_contexts.bin 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 904672 9627 /usr/lib64/libm.so.6 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 1293840 9789 /usr/lib64/libp11-kit.so.0.3.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 10007 /usr/lib64/libcrypto.so.3.0.7 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 1174 /etc/selinux/targeted/contexts/files/file_contexts.homedirs.bin 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 44784 9785 /usr/lib64/libffi.so.8.1.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 153600 9809 /usr/lib64/libgpg-error.so.0.32.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 28552 9804 /usr/lib64/libattr.so.1.1.2501 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 32528 9723 /usr/lib64/libcap-ng.so.0.0.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 1288504 9883 /usr/lib64/libgcrypt.so.20.4.0 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 40424 10956 /usr/lib64/libeconf.so.0.4.1 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 178728 9694 /usr/lib64/liblzma.so.5.2.5 2024-03-28T15:55:40.460 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 882376 9699 /usr/lib64/libzstd.so.1.5.1 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 144120 9867 /usr/lib64/liblz4.so.1.9.3 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 201808 9696 /usr/lib64/libcrypt.so.2.0.0 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 36296 9702 /usr/lib64/libcap.so.2.48 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 10126 /usr/lib64/libblkid.so.1.1.0 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 10156 /usr/lib64/libmount.so.1.1.0 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 3645472 49253 /usr/lib64/systemd/libsystemd-shared-252.so 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu DEL REG 8,1 9806 /usr/lib64/libacl.so.1.1.2301 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 107424 10180 /usr/lib64/libkmod.so.2.3.6 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 134624 9727 /usr/lib64/libaudit.so.1.0.0 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 69472 10965 /usr/lib64/libpam.so.0.85.1 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 2174088 49251 /usr/lib64/systemd/libsystemd-core-252.so 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 127680 9972 /usr/lib64/libseccomp.so.2.5.2 2024-03-28T15:55:40.461 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 108136 13107 /usr/lib64/libgcc_s-11-20230605.so.1 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu mem REG 0,19 0 19 /sys/fs/selinux/status 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 0r CHR 1,3 0t0 4 /dev/null 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 1u unix 0x0000000000000000 0t0 23246 type=STREAM (CONNECTED) 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 2u unix 0x0000000000000000 0t0 23246 type=STREAM (CONNECTED) 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 3u unix 0x0000000000000000 0t0 23274 type=DGRAM (CONNECTED) 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 4u a_inode 0,14 0 2053 [eventpoll:5,6,8,9,11,12,14,15,16,17,19,20,21] 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 5u a_inode 0,14 0 2053 [signalfd] 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 6r a_inode 0,14 0 2053 inotify 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 7r DIR 0,25 0 3393 /sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 8u a_inode 0,14 0 2053 [timerfd] 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 9u a_inode 0,14 0 2053 [eventpoll:10,13] 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 10r REG 0,20 0 23263 /proc/4831/mountinfo 2024-03-28T15:55:40.462 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 11r a_inode 0,14 0 2053 inotify 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 12u unix 0x0000000000000000 0t0 23296 /run/user/1000/bus type=STREAM (LISTEN) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 13r a_inode 0,14 0 2053 inotify 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 14r REG 0,20 0 4026532076 /proc/swaps 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 15u netlink 0t0 23278 KOBJECT_UEVENT 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 16u unix 0x0000000000000000 0t0 23283 /run/user/1000/systemd/notify type=DGRAM (UNCONNECTED) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 17u unix 0x0000000000000000 0t0 23284 type=DGRAM (CONNECTED) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 18u unix 0x0000000000000000 0t0 23285 type=DGRAM (CONNECTED) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 19u unix 0x0000000000000000 0t0 23286 /run/user/1000/systemd/private type=STREAM (LISTEN) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 20u unix 0x0000000000000000 0t0 23288 type=STREAM (CONNECTED) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 4831 ubuntu 21u a_inode 0,14 0 2053 [timerfd] 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 4833 ubuntu cwd unknown /proc/4833/cwd (readlink: Permission denied) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 4833 ubuntu rtd unknown /proc/4833/root (readlink: Permission denied) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 4833 ubuntu txt unknown /proc/4833/exe (readlink: Permission denied) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 4833 ubuntu NOFD /proc/4833/fd (opendir: Permission denied) 2024-03-28T15:55:40.463 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 root cwd unknown /proc/5076/cwd (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 root rtd unknown /proc/5076/root (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 root txt unknown /proc/5076/exe (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 root NOFD /proc/5076/fd (opendir: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5078 in:imjour root cwd unknown /proc/5076/task/5078/cwd (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5078 in:imjour root rtd unknown /proc/5076/task/5078/root (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5078 in:imjour root txt unknown /proc/5076/task/5078/exe (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5078 in:imjour root NOFD /proc/5076/task/5078/fd (opendir: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5079 rs:main root cwd unknown /proc/5076/task/5079/cwd (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5079 rs:main root rtd unknown /proc/5076/task/5079/root (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5079 rs:main root txt unknown /proc/5076/task/5079/exe (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:rsyslogd 5076 5079 rs:main root NOFD /proc/5076/task/5079/fd (opendir: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:tls-strp 15811 root cwd unknown /proc/15811/cwd (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:tls-strp 15811 root rtd unknown /proc/15811/root (readlink: Permission denied) 2024-03-28T15:55:40.464 INFO:tasks.workunit.client.0.smithi023.stdout:tls-strp 15811 root txt unknown /proc/15811/exe (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:tls-strp 15811 root NOFD /proc/15811/fd (opendir: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:smartd 15859 root cwd unknown /proc/15859/cwd (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:smartd 15859 root rtd unknown /proc/15859/root (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:smartd 15859 root txt unknown /proc/15859/exe (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:smartd 15859 root NOFD /proc/15859/fd (opendir: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21217 root cwd unknown /proc/21217/cwd (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21217 root rtd unknown /proc/21217/root (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21217 root txt unknown /proc/21217/exe (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21217 root NOFD /proc/21217/fd (opendir: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21246 root cwd unknown /proc/21246/cwd (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21246 root rtd unknown /proc/21246/root (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21246 root txt unknown /proc/21246/exe (readlink: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21246 root NOFD /proc/21246/fd (opendir: Permission denied) 2024-03-28T15:55:40.465 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21275 root cwd unknown /proc/21275/cwd (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21275 root rtd unknown /proc/21275/root (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21275 root txt unknown /proc/21275/exe (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21275 root NOFD /proc/21275/fd (opendir: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21304 root cwd unknown /proc/21304/cwd (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21304 root rtd unknown /proc/21304/root (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21304 root txt unknown /proc/21304/exe (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21304 root NOFD /proc/21304/fd (opendir: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21333 root cwd unknown /proc/21333/cwd (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21333 root rtd unknown /proc/21333/root (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21333 root txt unknown /proc/21333/exe (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:kdmflush/ 21333 root NOFD /proc/21333/fd (opendir: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:xfsalloc 21497 root cwd unknown /proc/21497/cwd (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:xfsalloc 21497 root rtd unknown /proc/21497/root (readlink: Permission denied) 2024-03-28T15:55:40.466 INFO:tasks.workunit.client.0.smithi023.stdout:xfsalloc 21497 root txt unknown /proc/21497/exe (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfsalloc 21497 root NOFD /proc/21497/fd (opendir: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs_mru_c 21498 root cwd unknown /proc/21498/cwd (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs_mru_c 21498 root rtd unknown /proc/21498/root (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs_mru_c 21498 root txt unknown /proc/21498/exe (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs_mru_c 21498 root NOFD /proc/21498/fd (opendir: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-buf/d 21499 root cwd unknown /proc/21499/cwd (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-buf/d 21499 root rtd unknown /proc/21499/root (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-buf/d 21499 root txt unknown /proc/21499/exe (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-buf/d 21499 root NOFD /proc/21499/fd (opendir: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-conv/ 21500 root cwd unknown /proc/21500/cwd (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-conv/ 21500 root rtd unknown /proc/21500/root (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-conv/ 21500 root txt unknown /proc/21500/exe (readlink: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-conv/ 21500 root NOFD /proc/21500/fd (opendir: Permission denied) 2024-03-28T15:55:40.467 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-recla 21501 root cwd unknown /proc/21501/cwd (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-recla 21501 root rtd unknown /proc/21501/root (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-recla 21501 root txt unknown /proc/21501/exe (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-recla 21501 root NOFD /proc/21501/fd (opendir: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-block 21502 root cwd unknown /proc/21502/cwd (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-block 21502 root rtd unknown /proc/21502/root (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-block 21502 root txt unknown /proc/21502/exe (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-block 21502 root NOFD /proc/21502/fd (opendir: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-inode 21503 root cwd unknown /proc/21503/cwd (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-inode 21503 root rtd unknown /proc/21503/root (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-inode 21503 root txt unknown /proc/21503/exe (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-inode 21503 root NOFD /proc/21503/fd (opendir: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-log/d 21504 root cwd unknown /proc/21504/cwd (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-log/d 21504 root rtd unknown /proc/21504/root (readlink: Permission denied) 2024-03-28T15:55:40.468 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-log/d 21504 root txt unknown /proc/21504/exe (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-log/d 21504 root NOFD /proc/21504/fd (opendir: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-cil/d 21505 root cwd unknown /proc/21505/cwd (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-cil/d 21505 root rtd unknown /proc/21505/root (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-cil/d 21505 root txt unknown /proc/21505/exe (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfs-cil/d 21505 root NOFD /proc/21505/fd (opendir: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfsaild/d 21506 root cwd unknown /proc/21506/cwd (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfsaild/d 21506 root rtd unknown /proc/21506/root (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfsaild/d 21506 root txt unknown /proc/21506/exe (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:xfsaild/d 21506 root NOFD /proc/21506/fd (opendir: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 21759 root cwd unknown /proc/21759/cwd (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 21759 root rtd unknown /proc/21759/root (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 21759 root txt unknown /proc/21759/exe (readlink: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 21759 root NOFD /proc/21759/fd (opendir: Permission denied) 2024-03-28T15:55:40.469 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23451 root cwd unknown /proc/23451/cwd (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23451 root rtd unknown /proc/23451/root (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23451 root txt unknown /proc/23451/exe (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23451 root NOFD /proc/23451/fd (opendir: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23454 ubuntu cwd unknown /proc/23454/cwd (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23454 ubuntu rtd unknown /proc/23454/root (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23454 ubuntu txt unknown /proc/23454/exe (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 23454 ubuntu NOFD /proc/23454/fd (opendir: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:chronyd 23514 chrony cwd unknown /proc/23514/cwd (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:chronyd 23514 chrony rtd unknown /proc/23514/root (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:chronyd 23514 chrony txt unknown /proc/23514/exe (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:chronyd 23514 chrony NOFD /proc/23514/fd (opendir: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 23664 root cwd unknown /proc/23664/cwd (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 23664 root rtd unknown /proc/23664/root (readlink: Permission denied) 2024-03-28T15:55:40.470 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 23664 root txt unknown /proc/23664/exe (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 23664 root NOFD /proc/23664/fd (opendir: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-cras 23818 ceph cwd unknown /proc/23818/cwd (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-cras 23818 ceph rtd unknown /proc/23818/root (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-cras 23818 ceph txt unknown /proc/23818/exe (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-cras 23818 ceph NOFD /proc/23818/fd (opendir: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 25652 root cwd unknown /proc/25652/cwd (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 25652 root rtd unknown /proc/25652/root (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 25652 root txt unknown /proc/25652/exe (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 25652 root NOFD /proc/25652/fd (opendir: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26857 root cwd unknown /proc/26857/cwd (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26857 root rtd unknown /proc/26857/root (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26857 root txt unknown /proc/26857/exe (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26857 root NOFD /proc/26857/fd (opendir: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26858 root cwd unknown /proc/26858/cwd (readlink: Permission denied) 2024-03-28T15:55:40.471 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26858 root rtd unknown /proc/26858/root (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26858 root txt unknown /proc/26858/exe (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 26858 root NOFD /proc/26858/fd (opendir: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 27255 root cwd unknown /proc/27255/cwd (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 27255 root rtd unknown /proc/27255/root (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 27255 root txt unknown /proc/27255/exe (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 27255 root NOFD /proc/27255/fd (opendir: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31425 root cwd unknown /proc/31425/cwd (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31425 root rtd unknown /proc/31425/root (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31425 root txt unknown /proc/31425/exe (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31425 root NOFD /proc/31425/fd (opendir: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31442 root cwd unknown /proc/31442/cwd (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31442 root rtd unknown /proc/31442/root (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31442 root txt unknown /proc/31442/exe (readlink: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31442 root NOFD /proc/31442/fd (opendir: Permission denied) 2024-03-28T15:55:40.472 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31443 root cwd unknown /proc/31443/cwd (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31443 root rtd unknown /proc/31443/root (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31443 root txt unknown /proc/31443/exe (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 31443 root NOFD /proc/31443/fd (opendir: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31460 root cwd unknown /proc/31460/cwd (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31460 root rtd unknown /proc/31460/root (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31460 root txt unknown /proc/31460/exe (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 31460 root NOFD /proc/31460/fd (opendir: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 31796 root cwd unknown /proc/31796/cwd (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 31796 root rtd unknown /proc/31796/root (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 31796 root txt unknown /proc/31796/exe (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 31796 root NOFD /proc/31796/fd (opendir: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 32221 root cwd unknown /proc/32221/cwd (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 32221 root rtd unknown /proc/32221/root (readlink: Permission denied) 2024-03-28T15:55:40.473 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 32221 root txt unknown /proc/32221/exe (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 32221 root NOFD /proc/32221/fd (opendir: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32444 root cwd unknown /proc/32444/cwd (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32444 root rtd unknown /proc/32444/root (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32444 root txt unknown /proc/32444/exe (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32444 root NOFD /proc/32444/fd (opendir: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32446 root cwd unknown /proc/32446/cwd (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32446 root rtd unknown /proc/32446/root (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32446 root txt unknown /proc/32446/exe (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32446 root NOFD /proc/32446/fd (opendir: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 ceph cwd unknown /proc/32448/cwd (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 ceph rtd unknown /proc/32448/root (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 ceph txt unknown /proc/32448/exe (readlink: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 ceph NOFD /proc/32448/fd (opendir: Permission denied) 2024-03-28T15:55:40.474 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32449 log ceph cwd unknown /proc/32448/task/32449/cwd (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32449 log ceph rtd unknown /proc/32448/task/32449/root (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32449 log ceph txt unknown /proc/32448/task/32449/exe (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32449 log ceph NOFD /proc/32448/task/32449/fd (opendir: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32450 service ceph cwd unknown /proc/32448/task/32450/cwd (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32450 service ceph rtd unknown /proc/32448/task/32450/root (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32450 service ceph txt unknown /proc/32448/task/32450/exe (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32450 service ceph NOFD /proc/32448/task/32450/fd (opendir: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32451 admin_soc ceph cwd unknown /proc/32448/task/32451/cwd (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32451 admin_soc ceph rtd unknown /proc/32448/task/32451/root (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32451 admin_soc ceph txt unknown /proc/32448/task/32451/exe (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32451 admin_soc ceph NOFD /proc/32448/task/32451/fd (opendir: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32452 signal_ha ceph cwd unknown /proc/32448/task/32452/cwd (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32452 signal_ha ceph rtd unknown /proc/32448/task/32452/root (readlink: Permission denied) 2024-03-28T15:55:40.475 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32452 signal_ha ceph txt unknown /proc/32448/task/32452/exe (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32452 signal_ha ceph NOFD /proc/32448/task/32452/fd (opendir: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32453 rocksdb:l ceph cwd unknown /proc/32448/task/32453/cwd (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32453 rocksdb:l ceph rtd unknown /proc/32448/task/32453/root (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32453 rocksdb:l ceph txt unknown /proc/32448/task/32453/exe (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32453 rocksdb:l ceph NOFD /proc/32448/task/32453/fd (opendir: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32454 rocksdb:h ceph cwd unknown /proc/32448/task/32454/cwd (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32454 rocksdb:h ceph rtd unknown /proc/32448/task/32454/root (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32454 rocksdb:h ceph txt unknown /proc/32448/task/32454/exe (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32454 rocksdb:h ceph NOFD /proc/32448/task/32454/fd (opendir: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32470 ceph-mon ceph cwd unknown /proc/32448/task/32470/cwd (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32470 ceph-mon ceph rtd unknown /proc/32448/task/32470/root (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32470 ceph-mon ceph txt unknown /proc/32448/task/32470/exe (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32470 ceph-mon ceph NOFD /proc/32448/task/32470/fd (opendir: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32471 fn_monsto ceph cwd unknown /proc/32448/task/32471/cwd (readlink: Permission denied) 2024-03-28T15:55:40.476 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32471 fn_monsto ceph rtd unknown /proc/32448/task/32471/root (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32471 fn_monsto ceph txt unknown /proc/32448/task/32471/exe (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32471 fn_monsto ceph NOFD /proc/32448/task/32471/fd (opendir: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32472 msgr-work ceph cwd unknown /proc/32448/task/32472/cwd (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32472 msgr-work ceph rtd unknown /proc/32448/task/32472/root (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32472 msgr-work ceph txt unknown /proc/32448/task/32472/exe (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32472 msgr-work ceph NOFD /proc/32448/task/32472/fd (opendir: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32473 msgr-work ceph cwd unknown /proc/32448/task/32473/cwd (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32473 msgr-work ceph rtd unknown /proc/32448/task/32473/root (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32473 msgr-work ceph txt unknown /proc/32448/task/32473/exe (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32473 msgr-work ceph NOFD /proc/32448/task/32473/fd (opendir: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32474 msgr-work ceph cwd unknown /proc/32448/task/32474/cwd (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32474 msgr-work ceph rtd unknown /proc/32448/task/32474/root (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32474 msgr-work ceph txt unknown /proc/32448/task/32474/exe (readlink: Permission denied) 2024-03-28T15:55:40.477 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32474 msgr-work ceph NOFD /proc/32448/task/32474/fd (opendir: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32475 OpHistory ceph cwd unknown /proc/32448/task/32475/cwd (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32475 OpHistory ceph rtd unknown /proc/32448/task/32475/root (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32475 OpHistory ceph txt unknown /proc/32448/task/32475/exe (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32475 OpHistory ceph NOFD /proc/32448/task/32475/fd (opendir: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32476 fin ceph cwd unknown /proc/32448/task/32476/cwd (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32476 fin ceph rtd unknown /proc/32448/task/32476/root (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32476 fin ceph txt unknown /proc/32448/task/32476/exe (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32476 fin ceph NOFD /proc/32448/task/32476/fd (opendir: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32477 safe_time ceph cwd unknown /proc/32448/task/32477/cwd (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32477 safe_time ceph rtd unknown /proc/32448/task/32477/root (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32477 safe_time ceph txt unknown /proc/32448/task/32477/exe (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32477 safe_time ceph NOFD /proc/32448/task/32477/fd (opendir: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32478 cpu_tp ceph cwd unknown /proc/32448/task/32478/cwd (readlink: Permission denied) 2024-03-28T15:55:40.478 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32478 cpu_tp ceph rtd unknown /proc/32448/task/32478/root (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32478 cpu_tp ceph txt unknown /proc/32448/task/32478/exe (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32478 cpu_tp ceph NOFD /proc/32448/task/32478/fd (opendir: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32479 cpu_tp ceph cwd unknown /proc/32448/task/32479/cwd (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32479 cpu_tp ceph rtd unknown /proc/32448/task/32479/root (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32479 cpu_tp ceph txt unknown /proc/32448/task/32479/exe (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32479 cpu_tp ceph NOFD /proc/32448/task/32479/fd (opendir: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32480 cpu_tp ceph cwd unknown /proc/32448/task/32480/cwd (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32480 cpu_tp ceph rtd unknown /proc/32448/task/32480/root (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32480 cpu_tp ceph txt unknown /proc/32448/task/32480/exe (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32480 cpu_tp ceph NOFD /proc/32448/task/32480/fd (opendir: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32481 cpu_tp ceph cwd unknown /proc/32448/task/32481/cwd (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32481 cpu_tp ceph rtd unknown /proc/32448/task/32481/root (readlink: Permission denied) 2024-03-28T15:55:40.479 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32481 cpu_tp ceph txt unknown /proc/32448/task/32481/exe (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32481 cpu_tp ceph NOFD /proc/32448/task/32481/fd (opendir: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32482 ms_dispat ceph cwd unknown /proc/32448/task/32482/cwd (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32482 ms_dispat ceph rtd unknown /proc/32448/task/32482/root (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32482 ms_dispat ceph txt unknown /proc/32448/task/32482/exe (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32482 ms_dispat ceph NOFD /proc/32448/task/32482/fd (opendir: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32483 ms_local ceph cwd unknown /proc/32448/task/32483/cwd (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32483 ms_local ceph rtd unknown /proc/32448/task/32483/root (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32483 ms_local ceph txt unknown /proc/32448/task/32483/exe (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32483 ms_local ceph NOFD /proc/32448/task/32483/fd (opendir: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32484 safe_time ceph cwd unknown /proc/32448/task/32484/cwd (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32484 safe_time ceph rtd unknown /proc/32448/task/32484/root (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32484 safe_time ceph txt unknown /proc/32448/task/32484/exe (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32484 safe_time ceph NOFD /proc/32448/task/32484/fd (opendir: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32485 ms_dispat ceph cwd unknown /proc/32448/task/32485/cwd (readlink: Permission denied) 2024-03-28T15:55:40.480 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32485 ms_dispat ceph rtd unknown /proc/32448/task/32485/root (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32485 ms_dispat ceph txt unknown /proc/32448/task/32485/exe (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32485 ms_dispat ceph NOFD /proc/32448/task/32485/fd (opendir: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32486 ms_local ceph cwd unknown /proc/32448/task/32486/cwd (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32486 ms_local ceph rtd unknown /proc/32448/task/32486/root (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32486 ms_local ceph txt unknown /proc/32448/task/32486/exe (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 32486 ms_local ceph NOFD /proc/32448/task/32486/fd (opendir: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 70964 rstore_co ceph cwd unknown /proc/32448/task/70964/cwd (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 70964 rstore_co ceph rtd unknown /proc/32448/task/70964/root (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 70964 rstore_co ceph txt unknown /proc/32448/task/70964/exe (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mon 32448 70964 rstore_co ceph NOFD /proc/32448/task/70964/fd (opendir: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32675 root cwd unknown /proc/32675/cwd (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32675 root rtd unknown /proc/32675/root (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32675 root txt unknown /proc/32675/exe (readlink: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 32675 root NOFD /proc/32675/fd (opendir: Permission denied) 2024-03-28T15:55:40.481 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32677 root cwd unknown /proc/32677/cwd (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32677 root rtd unknown /proc/32677/root (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32677 root txt unknown /proc/32677/exe (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 32677 root NOFD /proc/32677/fd (opendir: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 ceph cwd unknown /proc/32679/cwd (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 ceph rtd unknown /proc/32679/root (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 ceph txt unknown /proc/32679/exe (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 ceph NOFD /proc/32679/fd (opendir: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85874 log ceph cwd unknown /proc/32679/task/85874/cwd (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85874 log ceph rtd unknown /proc/32679/task/85874/root (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85874 log ceph txt unknown /proc/32679/task/85874/exe (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85874 log ceph NOFD /proc/32679/task/85874/fd (opendir: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85876 msgr-work ceph cwd unknown /proc/32679/task/85876/cwd (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85876 msgr-work ceph rtd unknown /proc/32679/task/85876/root (readlink: Permission denied) 2024-03-28T15:55:40.482 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85876 msgr-work ceph txt unknown /proc/32679/task/85876/exe (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85876 msgr-work ceph NOFD /proc/32679/task/85876/fd (opendir: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85877 msgr-work ceph cwd unknown /proc/32679/task/85877/cwd (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85877 msgr-work ceph rtd unknown /proc/32679/task/85877/root (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85877 msgr-work ceph txt unknown /proc/32679/task/85877/exe (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85877 msgr-work ceph NOFD /proc/32679/task/85877/fd (opendir: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85878 msgr-work ceph cwd unknown /proc/32679/task/85878/cwd (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85878 msgr-work ceph rtd unknown /proc/32679/task/85878/root (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85878 msgr-work ceph txt unknown /proc/32679/task/85878/exe (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85878 msgr-work ceph NOFD /proc/32679/task/85878/fd (opendir: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85882 service ceph cwd unknown /proc/32679/task/85882/cwd (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85882 service ceph rtd unknown /proc/32679/task/85882/root (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85882 service ceph txt unknown /proc/32679/task/85882/exe (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85882 service ceph NOFD /proc/32679/task/85882/fd (opendir: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85883 admin_soc ceph cwd unknown /proc/32679/task/85883/cwd (readlink: Permission denied) 2024-03-28T15:55:40.483 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85883 admin_soc ceph rtd unknown /proc/32679/task/85883/root (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85883 admin_soc ceph txt unknown /proc/32679/task/85883/exe (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85883 admin_soc ceph NOFD /proc/32679/task/85883/fd (opendir: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85884 ceph_time ceph cwd unknown /proc/32679/task/85884/cwd (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85884 ceph_time ceph rtd unknown /proc/32679/task/85884/root (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85884 ceph_time ceph txt unknown /proc/32679/task/85884/exe (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85884 ceph_time ceph NOFD /proc/32679/task/85884/fd (opendir: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85885 fn_anonym ceph cwd unknown /proc/32679/task/85885/cwd (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85885 fn_anonym ceph rtd unknown /proc/32679/task/85885/root (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85885 fn_anonym ceph txt unknown /proc/32679/task/85885/exe (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85885 fn_anonym ceph NOFD /proc/32679/task/85885/fd (opendir: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85886 signal_ha ceph cwd unknown /proc/32679/task/85886/cwd (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85886 signal_ha ceph rtd unknown /proc/32679/task/85886/root (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85886 signal_ha ceph txt unknown /proc/32679/task/85886/exe (readlink: Permission denied) 2024-03-28T15:55:40.484 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85886 signal_ha ceph NOFD /proc/32679/task/85886/fd (opendir: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85887 mgrsb-fin ceph cwd unknown /proc/32679/task/85887/cwd (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85887 mgrsb-fin ceph rtd unknown /proc/32679/task/85887/root (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85887 mgrsb-fin ceph txt unknown /proc/32679/task/85887/exe (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85887 mgrsb-fin ceph NOFD /proc/32679/task/85887/fd (opendir: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85888 ms_dispat ceph cwd unknown /proc/32679/task/85888/cwd (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85888 ms_dispat ceph rtd unknown /proc/32679/task/85888/root (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85888 ms_dispat ceph txt unknown /proc/32679/task/85888/exe (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85888 ms_dispat ceph NOFD /proc/32679/task/85888/fd (opendir: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85889 ms_local ceph cwd unknown /proc/32679/task/85889/cwd (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85889 ms_local ceph rtd unknown /proc/32679/task/85889/root (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85889 ms_local ceph txt unknown /proc/32679/task/85889/exe (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85889 ms_local ceph NOFD /proc/32679/task/85889/fd (opendir: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85890 io_contex ceph cwd unknown /proc/32679/task/85890/cwd (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85890 io_contex ceph rtd unknown /proc/32679/task/85890/root (readlink: Permission denied) 2024-03-28T15:55:40.485 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85890 io_contex ceph txt unknown /proc/32679/task/85890/exe (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85890 io_contex ceph NOFD /proc/32679/task/85890/fd (opendir: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85891 io_contex ceph cwd unknown /proc/32679/task/85891/cwd (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85891 io_contex ceph rtd unknown /proc/32679/task/85891/root (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85891 io_contex ceph txt unknown /proc/32679/task/85891/exe (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85891 io_contex ceph NOFD /proc/32679/task/85891/fd (opendir: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85892 safe_time ceph cwd unknown /proc/32679/task/85892/cwd (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85892 safe_time ceph rtd unknown /proc/32679/task/85892/root (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85892 safe_time ceph txt unknown /proc/32679/task/85892/exe (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85892 safe_time ceph NOFD /proc/32679/task/85892/fd (opendir: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85893 safe_time ceph cwd unknown /proc/32679/task/85893/cwd (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85893 safe_time ceph rtd unknown /proc/32679/task/85893/root (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85893 safe_time ceph txt unknown /proc/32679/task/85893/exe (readlink: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85893 safe_time ceph NOFD /proc/32679/task/85893/fd (opendir: Permission denied) 2024-03-28T15:55:40.486 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85894 safe_time ceph cwd unknown /proc/32679/task/85894/cwd (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85894 safe_time ceph rtd unknown /proc/32679/task/85894/root (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85894 safe_time ceph txt unknown /proc/32679/task/85894/exe (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85894 safe_time ceph NOFD /proc/32679/task/85894/fd (opendir: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85895 fn_anonym ceph cwd unknown /proc/32679/task/85895/cwd (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85895 fn_anonym ceph rtd unknown /proc/32679/task/85895/root (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85895 fn_anonym ceph txt unknown /proc/32679/task/85895/exe (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85895 fn_anonym ceph NOFD /proc/32679/task/85895/fd (opendir: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85896 flusher ceph cwd unknown /proc/32679/task/85896/cwd (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85896 flusher ceph rtd unknown /proc/32679/task/85896/root (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85896 flusher ceph txt unknown /proc/32679/task/85896/exe (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85896 flusher ceph NOFD /proc/32679/task/85896/fd (opendir: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85897 safe_time ceph cwd unknown /proc/32679/task/85897/cwd (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85897 safe_time ceph rtd unknown /proc/32679/task/85897/root (readlink: Permission denied) 2024-03-28T15:55:40.487 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85897 safe_time ceph txt unknown /proc/32679/task/85897/exe (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 85897 safe_time ceph NOFD /proc/32679/task/85897/fd (opendir: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86072 mgr-fin ceph cwd unknown /proc/32679/task/86072/cwd (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86072 mgr-fin ceph rtd unknown /proc/32679/task/86072/root (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86072 mgr-fin ceph txt unknown /proc/32679/task/86072/exe (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86072 mgr-fin ceph NOFD /proc/32679/task/86072/fd (opendir: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86073 ms_dispat ceph cwd unknown /proc/32679/task/86073/cwd (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86073 ms_dispat ceph rtd unknown /proc/32679/task/86073/root (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86073 ms_dispat ceph txt unknown /proc/32679/task/86073/exe (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86073 ms_dispat ceph NOFD /proc/32679/task/86073/fd (opendir: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86074 ms_local ceph cwd unknown /proc/32679/task/86074/cwd (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86074 ms_local ceph rtd unknown /proc/32679/task/86074/root (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86074 ms_local ceph txt unknown /proc/32679/task/86074/exe (readlink: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86074 ms_local ceph NOFD /proc/32679/task/86074/fd (opendir: Permission denied) 2024-03-28T15:55:40.488 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86075 safe_time ceph cwd unknown /proc/32679/task/86075/cwd (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86075 safe_time ceph rtd unknown /proc/32679/task/86075/root (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86075 safe_time ceph txt unknown /proc/32679/task/86075/exe (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86075 safe_time ceph NOFD /proc/32679/task/86075/fd (opendir: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86076 cmdfin ceph cwd unknown /proc/32679/task/86076/cwd (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86076 cmdfin ceph rtd unknown /proc/32679/task/86076/root (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86076 cmdfin ceph txt unknown /proc/32679/task/86076/exe (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86076 cmdfin ceph NOFD /proc/32679/task/86076/fd (opendir: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86080 io_contex ceph cwd unknown /proc/32679/task/86080/cwd (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86080 io_contex ceph rtd unknown /proc/32679/task/86080/root (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86080 io_contex ceph txt unknown /proc/32679/task/86080/exe (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86080 io_contex ceph NOFD /proc/32679/task/86080/fd (opendir: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86081 io_contex ceph cwd unknown /proc/32679/task/86081/cwd (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86081 io_contex ceph rtd unknown /proc/32679/task/86081/root (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86081 io_contex ceph txt unknown /proc/32679/task/86081/exe (readlink: Permission denied) 2024-03-28T15:55:40.489 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86081 io_contex ceph NOFD /proc/32679/task/86081/fd (opendir: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86082 ceph_time ceph cwd unknown /proc/32679/task/86082/cwd (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86082 ceph_time ceph rtd unknown /proc/32679/task/86082/root (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86082 ceph_time ceph txt unknown /proc/32679/task/86082/exe (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86082 ceph_time ceph NOFD /proc/32679/task/86082/fd (opendir: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86083 ms_dispat ceph cwd unknown /proc/32679/task/86083/cwd (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86083 ms_dispat ceph rtd unknown /proc/32679/task/86083/root (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86083 ms_dispat ceph txt unknown /proc/32679/task/86083/exe (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86083 ms_dispat ceph NOFD /proc/32679/task/86083/fd (opendir: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86084 ms_local ceph cwd unknown /proc/32679/task/86084/cwd (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86084 ms_local ceph rtd unknown /proc/32679/task/86084/root (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86084 ms_local ceph txt unknown /proc/32679/task/86084/exe (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86084 ms_local ceph NOFD /proc/32679/task/86084/fd (opendir: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86085 safe_time ceph cwd unknown /proc/32679/task/86085/cwd (readlink: Permission denied) 2024-03-28T15:55:40.490 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86085 safe_time ceph rtd unknown /proc/32679/task/86085/root (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86085 safe_time ceph txt unknown /proc/32679/task/86085/exe (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86085 safe_time ceph NOFD /proc/32679/task/86085/fd (opendir: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86086 safe_time ceph cwd unknown /proc/32679/task/86086/cwd (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86086 safe_time ceph rtd unknown /proc/32679/task/86086/root (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86086 safe_time ceph txt unknown /proc/32679/task/86086/exe (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86086 safe_time ceph NOFD /proc/32679/task/86086/fd (opendir: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86087 balancer ceph cwd unknown /proc/32679/task/86087/cwd (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86087 balancer ceph rtd unknown /proc/32679/task/86087/root (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86087 balancer ceph txt unknown /proc/32679/task/86087/exe (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86087 balancer ceph NOFD /proc/32679/task/86087/fd (opendir: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86088 m-fin-bal ceph cwd unknown /proc/32679/task/86088/cwd (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86088 m-fin-bal ceph rtd unknown /proc/32679/task/86088/root (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86088 m-fin-bal ceph txt unknown /proc/32679/task/86088/exe (readlink: Permission denied) 2024-03-28T15:55:40.491 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86088 m-fin-bal ceph NOFD /proc/32679/task/86088/fd (opendir: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86089 mgr-fin ceph cwd unknown /proc/32679/task/86089/cwd (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86089 mgr-fin ceph rtd unknown /proc/32679/task/86089/root (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86089 mgr-fin ceph txt unknown /proc/32679/task/86089/exe (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86089 mgr-fin ceph NOFD /proc/32679/task/86089/fd (opendir: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86090 mgr-fin ceph cwd unknown /proc/32679/task/86090/cwd (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86090 mgr-fin ceph rtd unknown /proc/32679/task/86090/root (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86090 mgr-fin ceph txt unknown /proc/32679/task/86090/exe (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86090 mgr-fin ceph NOFD /proc/32679/task/86090/fd (opendir: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86091 mgr-fin ceph cwd unknown /proc/32679/task/86091/cwd (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86091 mgr-fin ceph rtd unknown /proc/32679/task/86091/root (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86091 mgr-fin ceph txt unknown /proc/32679/task/86091/exe (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86091 mgr-fin ceph NOFD /proc/32679/task/86091/fd (opendir: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86092 mgr-fin ceph cwd unknown /proc/32679/task/86092/cwd (readlink: Permission denied) 2024-03-28T15:55:40.492 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86092 mgr-fin ceph rtd unknown /proc/32679/task/86092/root (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86092 mgr-fin ceph txt unknown /proc/32679/task/86092/exe (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86092 mgr-fin ceph NOFD /proc/32679/task/86092/fd (opendir: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86093 mgr-fin ceph cwd unknown /proc/32679/task/86093/cwd (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86093 mgr-fin ceph rtd unknown /proc/32679/task/86093/root (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86093 mgr-fin ceph txt unknown /proc/32679/task/86093/exe (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86093 mgr-fin ceph NOFD /proc/32679/task/86093/fd (opendir: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86094 mgr-fin ceph cwd unknown /proc/32679/task/86094/cwd (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86094 mgr-fin ceph rtd unknown /proc/32679/task/86094/root (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86094 mgr-fin ceph txt unknown /proc/32679/task/86094/exe (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86094 mgr-fin ceph NOFD /proc/32679/task/86094/fd (opendir: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86095 mgr-fin ceph cwd unknown /proc/32679/task/86095/cwd (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86095 mgr-fin ceph rtd unknown /proc/32679/task/86095/root (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86095 mgr-fin ceph txt unknown /proc/32679/task/86095/exe (readlink: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86095 mgr-fin ceph NOFD /proc/32679/task/86095/fd (opendir: Permission denied) 2024-03-28T15:55:40.493 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86096 mgr-fin ceph cwd unknown /proc/32679/task/86096/cwd (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86096 mgr-fin ceph rtd unknown /proc/32679/task/86096/root (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86096 mgr-fin ceph txt unknown /proc/32679/task/86096/exe (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86096 mgr-fin ceph NOFD /proc/32679/task/86096/fd (opendir: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86097 mgr-fin ceph cwd unknown /proc/32679/task/86097/cwd (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86097 mgr-fin ceph rtd unknown /proc/32679/task/86097/root (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86097 mgr-fin ceph txt unknown /proc/32679/task/86097/exe (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86097 mgr-fin ceph NOFD /proc/32679/task/86097/fd (opendir: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86098 mgr-fin ceph cwd unknown /proc/32679/task/86098/cwd (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86098 mgr-fin ceph rtd unknown /proc/32679/task/86098/root (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86098 mgr-fin ceph txt unknown /proc/32679/task/86098/exe (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86098 mgr-fin ceph NOFD /proc/32679/task/86098/fd (opendir: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86099 mgr-fin ceph cwd unknown /proc/32679/task/86099/cwd (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86099 mgr-fin ceph rtd unknown /proc/32679/task/86099/root (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86099 mgr-fin ceph txt unknown /proc/32679/task/86099/exe (readlink: Permission denied) 2024-03-28T15:55:40.494 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86099 mgr-fin ceph NOFD /proc/32679/task/86099/fd (opendir: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86100 mgr-fin ceph cwd unknown /proc/32679/task/86100/cwd (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86100 mgr-fin ceph rtd unknown /proc/32679/task/86100/root (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86100 mgr-fin ceph txt unknown /proc/32679/task/86100/exe (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86100 mgr-fin ceph NOFD /proc/32679/task/86100/fd (opendir: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86101 mgr-fin ceph cwd unknown /proc/32679/task/86101/cwd (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86101 mgr-fin ceph rtd unknown /proc/32679/task/86101/root (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86101 mgr-fin ceph txt unknown /proc/32679/task/86101/exe (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86101 mgr-fin ceph NOFD /proc/32679/task/86101/fd (opendir: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86102 mgr-fin ceph cwd unknown /proc/32679/task/86102/cwd (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86102 mgr-fin ceph rtd unknown /proc/32679/task/86102/root (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86102 mgr-fin ceph txt unknown /proc/32679/task/86102/exe (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86102 mgr-fin ceph NOFD /proc/32679/task/86102/fd (opendir: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86103 mgr-fin ceph cwd unknown /proc/32679/task/86103/cwd (readlink: Permission denied) 2024-03-28T15:55:40.495 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86103 mgr-fin ceph rtd unknown /proc/32679/task/86103/root (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86103 mgr-fin ceph txt unknown /proc/32679/task/86103/exe (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86103 mgr-fin ceph NOFD /proc/32679/task/86103/fd (opendir: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86104 mgr-fin ceph cwd unknown /proc/32679/task/86104/cwd (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86104 mgr-fin ceph rtd unknown /proc/32679/task/86104/root (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86104 mgr-fin ceph txt unknown /proc/32679/task/86104/exe (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86104 mgr-fin ceph NOFD /proc/32679/task/86104/fd (opendir: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86105 cephadm ceph cwd unknown /proc/32679/task/86105/cwd (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86105 cephadm ceph rtd unknown /proc/32679/task/86105/root (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86105 cephadm ceph txt unknown /proc/32679/task/86105/exe (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86105 cephadm ceph NOFD /proc/32679/task/86105/fd (opendir: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86106 m-fin-cep ceph cwd unknown /proc/32679/task/86106/cwd (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86106 m-fin-cep ceph rtd unknown /proc/32679/task/86106/root (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86106 m-fin-cep ceph txt unknown /proc/32679/task/86106/exe (readlink: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86106 m-fin-cep ceph NOFD /proc/32679/task/86106/fd (opendir: Permission denied) 2024-03-28T15:55:40.496 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86107 mgr-fin ceph cwd unknown /proc/32679/task/86107/cwd (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86107 mgr-fin ceph rtd unknown /proc/32679/task/86107/root (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86107 mgr-fin ceph txt unknown /proc/32679/task/86107/exe (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86107 mgr-fin ceph NOFD /proc/32679/task/86107/fd (opendir: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86108 cephadm ceph cwd unknown /proc/32679/task/86108/cwd (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86108 cephadm ceph rtd unknown /proc/32679/task/86108/root (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86108 cephadm ceph txt unknown /proc/32679/task/86108/exe (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86108 cephadm ceph NOFD /proc/32679/task/86108/fd (opendir: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86109 mgr-fin ceph cwd unknown /proc/32679/task/86109/cwd (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86109 mgr-fin ceph rtd unknown /proc/32679/task/86109/root (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86109 mgr-fin ceph txt unknown /proc/32679/task/86109/exe (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86109 mgr-fin ceph NOFD /proc/32679/task/86109/fd (opendir: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86110 mgr-fin ceph cwd unknown /proc/32679/task/86110/cwd (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86110 mgr-fin ceph rtd unknown /proc/32679/task/86110/root (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86110 mgr-fin ceph txt unknown /proc/32679/task/86110/exe (readlink: Permission denied) 2024-03-28T15:55:40.497 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86110 mgr-fin ceph NOFD /proc/32679/task/86110/fd (opendir: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86111 mgr-fin ceph cwd unknown /proc/32679/task/86111/cwd (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86111 mgr-fin ceph rtd unknown /proc/32679/task/86111/root (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86111 mgr-fin ceph txt unknown /proc/32679/task/86111/exe (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86111 mgr-fin ceph NOFD /proc/32679/task/86111/fd (opendir: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86112 mgr-fin ceph cwd unknown /proc/32679/task/86112/cwd (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86112 mgr-fin ceph rtd unknown /proc/32679/task/86112/root (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86112 mgr-fin ceph txt unknown /proc/32679/task/86112/exe (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86112 mgr-fin ceph NOFD /proc/32679/task/86112/fd (opendir: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86113 crash ceph cwd unknown /proc/32679/task/86113/cwd (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86113 crash ceph rtd unknown /proc/32679/task/86113/root (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86113 crash ceph txt unknown /proc/32679/task/86113/exe (readlink: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86113 crash ceph NOFD /proc/32679/task/86113/fd (opendir: Permission denied) 2024-03-28T15:55:40.498 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86114 m-fin-cra ceph cwd unknown /proc/32679/task/86114/cwd (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86114 m-fin-cra ceph rtd unknown /proc/32679/task/86114/root (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86114 m-fin-cra ceph txt unknown /proc/32679/task/86114/exe (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86114 m-fin-cra ceph NOFD /proc/32679/task/86114/fd (opendir: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86115 mgr-fin ceph cwd unknown /proc/32679/task/86115/cwd (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86115 mgr-fin ceph rtd unknown /proc/32679/task/86115/root (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86115 mgr-fin ceph txt unknown /proc/32679/task/86115/exe (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86115 mgr-fin ceph NOFD /proc/32679/task/86115/fd (opendir: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86116 dashboard ceph cwd unknown /proc/32679/task/86116/cwd (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86116 dashboard ceph rtd unknown /proc/32679/task/86116/root (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86116 dashboard ceph txt unknown /proc/32679/task/86116/exe (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86116 dashboard ceph NOFD /proc/32679/task/86116/fd (opendir: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86117 m-fin-das ceph cwd unknown /proc/32679/task/86117/cwd (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86117 m-fin-das ceph rtd unknown /proc/32679/task/86117/root (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86117 m-fin-das ceph txt unknown /proc/32679/task/86117/exe (readlink: Permission denied) 2024-03-28T15:55:40.499 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86117 m-fin-das ceph NOFD /proc/32679/task/86117/fd (opendir: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86118 devicehea ceph cwd unknown /proc/32679/task/86118/cwd (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86118 devicehea ceph rtd unknown /proc/32679/task/86118/root (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86118 devicehea ceph txt unknown /proc/32679/task/86118/exe (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86118 devicehea ceph NOFD /proc/32679/task/86118/fd (opendir: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86119 m-fin-dev ceph cwd unknown /proc/32679/task/86119/cwd (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86119 m-fin-dev ceph rtd unknown /proc/32679/task/86119/root (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86119 m-fin-dev ceph txt unknown /proc/32679/task/86119/exe (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86119 m-fin-dev ceph NOFD /proc/32679/task/86119/fd (opendir: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86120 mgr-fin ceph cwd unknown /proc/32679/task/86120/cwd (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86120 mgr-fin ceph rtd unknown /proc/32679/task/86120/root (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86120 mgr-fin ceph txt unknown /proc/32679/task/86120/exe (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86120 mgr-fin ceph NOFD /proc/32679/task/86120/fd (opendir: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86121 mgr-fin ceph cwd unknown /proc/32679/task/86121/cwd (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86121 mgr-fin ceph rtd unknown /proc/32679/task/86121/root (readlink: Permission denied) 2024-03-28T15:55:40.500 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86121 mgr-fin ceph txt unknown /proc/32679/task/86121/exe (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86121 mgr-fin ceph NOFD /proc/32679/task/86121/fd (opendir: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86123 m-fin-ios ceph cwd unknown /proc/32679/task/86123/cwd (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86123 m-fin-ios ceph rtd unknown /proc/32679/task/86123/root (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86123 m-fin-ios ceph txt unknown /proc/32679/task/86123/exe (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86123 m-fin-ios ceph NOFD /proc/32679/task/86123/fd (opendir: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86124 mgr-fin ceph cwd unknown /proc/32679/task/86124/cwd (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86124 mgr-fin ceph rtd unknown /proc/32679/task/86124/root (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86124 mgr-fin ceph txt unknown /proc/32679/task/86124/exe (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86124 mgr-fin ceph NOFD /proc/32679/task/86124/fd (opendir: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86126 m-fin-nfs ceph cwd unknown /proc/32679/task/86126/cwd (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86126 m-fin-nfs ceph rtd unknown /proc/32679/task/86126/root (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86126 m-fin-nfs ceph txt unknown /proc/32679/task/86126/exe (readlink: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86126 m-fin-nfs ceph NOFD /proc/32679/task/86126/fd (opendir: Permission denied) 2024-03-28T15:55:40.501 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86127 mgr-fin ceph cwd unknown /proc/32679/task/86127/cwd (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86127 mgr-fin ceph rtd unknown /proc/32679/task/86127/root (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86127 mgr-fin ceph txt unknown /proc/32679/task/86127/exe (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86127 mgr-fin ceph NOFD /proc/32679/task/86127/fd (opendir: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86128 mgr-fin ceph cwd unknown /proc/32679/task/86128/cwd (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86128 mgr-fin ceph rtd unknown /proc/32679/task/86128/root (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86128 mgr-fin ceph txt unknown /proc/32679/task/86128/exe (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86128 mgr-fin ceph NOFD /proc/32679/task/86128/fd (opendir: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86130 m-fin-orc ceph cwd unknown /proc/32679/task/86130/cwd (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86130 m-fin-orc ceph rtd unknown /proc/32679/task/86130/root (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86130 m-fin-orc ceph txt unknown /proc/32679/task/86130/exe (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86130 m-fin-orc ceph NOFD /proc/32679/task/86130/fd (opendir: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86131 mgr-fin ceph cwd unknown /proc/32679/task/86131/cwd (readlink: Permission denied) 2024-03-28T15:55:40.502 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86131 mgr-fin ceph rtd unknown /proc/32679/task/86131/root (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86131 mgr-fin ceph txt unknown /proc/32679/task/86131/exe (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86131 mgr-fin ceph NOFD /proc/32679/task/86131/fd (opendir: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86132 pg_autosc ceph cwd unknown /proc/32679/task/86132/cwd (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86132 pg_autosc ceph rtd unknown /proc/32679/task/86132/root (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86132 pg_autosc ceph txt unknown /proc/32679/task/86132/exe (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86132 pg_autosc ceph NOFD /proc/32679/task/86132/fd (opendir: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86133 m-fin-pg_ ceph cwd unknown /proc/32679/task/86133/cwd (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86133 m-fin-pg_ ceph rtd unknown /proc/32679/task/86133/root (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86133 m-fin-pg_ ceph txt unknown /proc/32679/task/86133/exe (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86133 m-fin-pg_ ceph NOFD /proc/32679/task/86133/fd (opendir: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86134 progress ceph cwd unknown /proc/32679/task/86134/cwd (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86134 progress ceph rtd unknown /proc/32679/task/86134/root (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86134 progress ceph txt unknown /proc/32679/task/86134/exe (readlink: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86134 progress ceph NOFD /proc/32679/task/86134/fd (opendir: Permission denied) 2024-03-28T15:55:40.503 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86135 m-fin-pro ceph cwd unknown /proc/32679/task/86135/cwd (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86135 m-fin-pro ceph rtd unknown /proc/32679/task/86135/root (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86135 m-fin-pro ceph txt unknown /proc/32679/task/86135/exe (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86135 m-fin-pro ceph NOFD /proc/32679/task/86135/fd (opendir: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86139 io_contex ceph cwd unknown /proc/32679/task/86139/cwd (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86139 io_contex ceph rtd unknown /proc/32679/task/86139/root (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86139 io_contex ceph txt unknown /proc/32679/task/86139/exe (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86139 io_contex ceph NOFD /proc/32679/task/86139/fd (opendir: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86140 io_contex ceph cwd unknown /proc/32679/task/86140/cwd (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86140 io_contex ceph rtd unknown /proc/32679/task/86140/root (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86140 io_contex ceph txt unknown /proc/32679/task/86140/exe (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86140 io_contex ceph NOFD /proc/32679/task/86140/fd (opendir: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86141 ceph_time ceph cwd unknown /proc/32679/task/86141/cwd (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86141 ceph_time ceph rtd unknown /proc/32679/task/86141/root (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86141 ceph_time ceph txt unknown /proc/32679/task/86141/exe (readlink: Permission denied) 2024-03-28T15:55:40.504 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86141 ceph_time ceph NOFD /proc/32679/task/86141/fd (opendir: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86142 ms_dispat ceph cwd unknown /proc/32679/task/86142/cwd (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86142 ms_dispat ceph rtd unknown /proc/32679/task/86142/root (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86142 ms_dispat ceph txt unknown /proc/32679/task/86142/exe (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86142 ms_dispat ceph NOFD /proc/32679/task/86142/fd (opendir: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86143 ms_local ceph cwd unknown /proc/32679/task/86143/cwd (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86143 ms_local ceph rtd unknown /proc/32679/task/86143/root (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86143 ms_local ceph txt unknown /proc/32679/task/86143/exe (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86143 ms_local ceph NOFD /proc/32679/task/86143/fd (opendir: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86144 safe_time ceph cwd unknown /proc/32679/task/86144/cwd (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86144 safe_time ceph rtd unknown /proc/32679/task/86144/root (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86144 safe_time ceph txt unknown /proc/32679/task/86144/exe (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86144 safe_time ceph NOFD /proc/32679/task/86144/fd (opendir: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86145 safe_time ceph cwd unknown /proc/32679/task/86145/cwd (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86145 safe_time ceph rtd unknown /proc/32679/task/86145/root (readlink: Permission denied) 2024-03-28T15:55:40.505 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86145 safe_time ceph txt unknown /proc/32679/task/86145/exe (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86145 safe_time ceph NOFD /proc/32679/task/86145/fd (opendir: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86146 mgr-fin ceph cwd unknown /proc/32679/task/86146/cwd (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86146 mgr-fin ceph rtd unknown /proc/32679/task/86146/root (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86146 mgr-fin ceph txt unknown /proc/32679/task/86146/exe (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86146 mgr-fin ceph NOFD /proc/32679/task/86146/fd (opendir: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86148 m-fin-rbd ceph cwd unknown /proc/32679/task/86148/cwd (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86148 m-fin-rbd ceph rtd unknown /proc/32679/task/86148/root (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86148 m-fin-rbd ceph txt unknown /proc/32679/task/86148/exe (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86148 m-fin-rbd ceph NOFD /proc/32679/task/86148/fd (opendir: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86149 restful ceph cwd unknown /proc/32679/task/86149/cwd (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86149 restful ceph rtd unknown /proc/32679/task/86149/root (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86149 restful ceph txt unknown /proc/32679/task/86149/exe (readlink: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86149 restful ceph NOFD /proc/32679/task/86149/fd (opendir: Permission denied) 2024-03-28T15:55:40.506 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86150 m-fin-res ceph cwd unknown /proc/32679/task/86150/cwd (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86150 m-fin-res ceph rtd unknown /proc/32679/task/86150/root (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86150 m-fin-res ceph txt unknown /proc/32679/task/86150/exe (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86150 m-fin-res ceph NOFD /proc/32679/task/86150/fd (opendir: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86151 mgr-fin ceph cwd unknown /proc/32679/task/86151/cwd (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86151 mgr-fin ceph rtd unknown /proc/32679/task/86151/root (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86151 mgr-fin ceph txt unknown /proc/32679/task/86151/exe (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86151 mgr-fin ceph NOFD /proc/32679/task/86151/fd (opendir: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86152 mgr-fin ceph cwd unknown /proc/32679/task/86152/cwd (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86152 mgr-fin ceph rtd unknown /proc/32679/task/86152/root (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86152 mgr-fin ceph txt unknown /proc/32679/task/86152/exe (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86152 mgr-fin ceph NOFD /proc/32679/task/86152/fd (opendir: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86153 mgr-fin ceph cwd unknown /proc/32679/task/86153/cwd (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86153 mgr-fin ceph rtd unknown /proc/32679/task/86153/root (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86153 mgr-fin ceph txt unknown /proc/32679/task/86153/exe (readlink: Permission denied) 2024-03-28T15:55:40.507 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86153 mgr-fin ceph NOFD /proc/32679/task/86153/fd (opendir: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86154 mgr-fin ceph cwd unknown /proc/32679/task/86154/cwd (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86154 mgr-fin ceph rtd unknown /proc/32679/task/86154/root (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86154 mgr-fin ceph txt unknown /proc/32679/task/86154/exe (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86154 mgr-fin ceph NOFD /proc/32679/task/86154/fd (opendir: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86155 mgr-fin ceph cwd unknown /proc/32679/task/86155/cwd (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86155 mgr-fin ceph rtd unknown /proc/32679/task/86155/root (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86155 mgr-fin ceph txt unknown /proc/32679/task/86155/exe (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86155 mgr-fin ceph NOFD /proc/32679/task/86155/fd (opendir: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86156 mgr-fin ceph cwd unknown /proc/32679/task/86156/cwd (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86156 mgr-fin ceph rtd unknown /proc/32679/task/86156/root (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86156 mgr-fin ceph txt unknown /proc/32679/task/86156/exe (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86156 mgr-fin ceph NOFD /proc/32679/task/86156/fd (opendir: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86160 io_contex ceph cwd unknown /proc/32679/task/86160/cwd (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86160 io_contex ceph rtd unknown /proc/32679/task/86160/root (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86160 io_contex ceph txt unknown /proc/32679/task/86160/exe (readlink: Permission denied) 2024-03-28T15:55:40.508 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86160 io_contex ceph NOFD /proc/32679/task/86160/fd (opendir: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86161 io_contex ceph cwd unknown /proc/32679/task/86161/cwd (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86161 io_contex ceph rtd unknown /proc/32679/task/86161/root (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86161 io_contex ceph txt unknown /proc/32679/task/86161/exe (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86161 io_contex ceph NOFD /proc/32679/task/86161/fd (opendir: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86162 ceph_time ceph cwd unknown /proc/32679/task/86162/cwd (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86162 ceph_time ceph rtd unknown /proc/32679/task/86162/root (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86162 ceph_time ceph txt unknown /proc/32679/task/86162/exe (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86162 ceph_time ceph NOFD /proc/32679/task/86162/fd (opendir: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86163 ms_dispat ceph cwd unknown /proc/32679/task/86163/cwd (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86163 ms_dispat ceph rtd unknown /proc/32679/task/86163/root (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86163 ms_dispat ceph txt unknown /proc/32679/task/86163/exe (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86163 ms_dispat ceph NOFD /proc/32679/task/86163/fd (opendir: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86164 ms_local ceph cwd unknown /proc/32679/task/86164/cwd (readlink: Permission denied) 2024-03-28T15:55:40.509 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86164 ms_local ceph rtd unknown /proc/32679/task/86164/root (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86164 ms_local ceph txt unknown /proc/32679/task/86164/exe (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86164 ms_local ceph NOFD /proc/32679/task/86164/fd (opendir: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86165 safe_time ceph cwd unknown /proc/32679/task/86165/cwd (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86165 safe_time ceph rtd unknown /proc/32679/task/86165/root (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86165 safe_time ceph txt unknown /proc/32679/task/86165/exe (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86165 safe_time ceph NOFD /proc/32679/task/86165/fd (opendir: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86166 safe_time ceph cwd unknown /proc/32679/task/86166/cwd (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86166 safe_time ceph rtd unknown /proc/32679/task/86166/root (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86166 safe_time ceph txt unknown /proc/32679/task/86166/exe (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86166 safe_time ceph NOFD /proc/32679/task/86166/fd (opendir: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86170 m-fin-sna ceph cwd unknown /proc/32679/task/86170/cwd (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86170 m-fin-sna ceph rtd unknown /proc/32679/task/86170/root (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86170 m-fin-sna ceph txt unknown /proc/32679/task/86170/exe (readlink: Permission denied) 2024-03-28T15:55:40.510 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86170 m-fin-sna ceph NOFD /proc/32679/task/86170/fd (opendir: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86172 m-fin-sta ceph cwd unknown /proc/32679/task/86172/cwd (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86172 m-fin-sta ceph rtd unknown /proc/32679/task/86172/root (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86172 m-fin-sta ceph txt unknown /proc/32679/task/86172/exe (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86172 m-fin-sta ceph NOFD /proc/32679/task/86172/fd (opendir: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86173 telemetry ceph cwd unknown /proc/32679/task/86173/cwd (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86173 telemetry ceph rtd unknown /proc/32679/task/86173/root (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86173 telemetry ceph txt unknown /proc/32679/task/86173/exe (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86173 telemetry ceph NOFD /proc/32679/task/86173/fd (opendir: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86174 m-fin-tel ceph cwd unknown /proc/32679/task/86174/cwd (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86174 m-fin-tel ceph rtd unknown /proc/32679/task/86174/root (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86174 m-fin-tel ceph txt unknown /proc/32679/task/86174/exe (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86174 m-fin-tel ceph NOFD /proc/32679/task/86174/fd (opendir: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86175 mgr-fin ceph cwd unknown /proc/32679/task/86175/cwd (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86175 mgr-fin ceph rtd unknown /proc/32679/task/86175/root (readlink: Permission denied) 2024-03-28T15:55:40.511 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86175 mgr-fin ceph txt unknown /proc/32679/task/86175/exe (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86175 mgr-fin ceph NOFD /proc/32679/task/86175/fd (opendir: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86179 io_contex ceph cwd unknown /proc/32679/task/86179/cwd (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86179 io_contex ceph rtd unknown /proc/32679/task/86179/root (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86179 io_contex ceph txt unknown /proc/32679/task/86179/exe (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86179 io_contex ceph NOFD /proc/32679/task/86179/fd (opendir: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86180 io_contex ceph cwd unknown /proc/32679/task/86180/cwd (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86180 io_contex ceph rtd unknown /proc/32679/task/86180/root (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86180 io_contex ceph txt unknown /proc/32679/task/86180/exe (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86180 io_contex ceph NOFD /proc/32679/task/86180/fd (opendir: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86181 ceph_time ceph cwd unknown /proc/32679/task/86181/cwd (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86181 ceph_time ceph rtd unknown /proc/32679/task/86181/root (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86181 ceph_time ceph txt unknown /proc/32679/task/86181/exe (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86181 ceph_time ceph NOFD /proc/32679/task/86181/fd (opendir: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86182 ms_dispat ceph cwd unknown /proc/32679/task/86182/cwd (readlink: Permission denied) 2024-03-28T15:55:40.512 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86182 ms_dispat ceph rtd unknown /proc/32679/task/86182/root (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86182 ms_dispat ceph txt unknown /proc/32679/task/86182/exe (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86182 ms_dispat ceph NOFD /proc/32679/task/86182/fd (opendir: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86183 ms_local ceph cwd unknown /proc/32679/task/86183/cwd (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86183 ms_local ceph rtd unknown /proc/32679/task/86183/root (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86183 ms_local ceph txt unknown /proc/32679/task/86183/exe (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86183 ms_local ceph NOFD /proc/32679/task/86183/fd (opendir: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86184 safe_time ceph cwd unknown /proc/32679/task/86184/cwd (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86184 safe_time ceph rtd unknown /proc/32679/task/86184/root (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86184 safe_time ceph txt unknown /proc/32679/task/86184/exe (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86184 safe_time ceph NOFD /proc/32679/task/86184/fd (opendir: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86185 safe_time ceph cwd unknown /proc/32679/task/86185/cwd (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86185 safe_time ceph rtd unknown /proc/32679/task/86185/root (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86185 safe_time ceph txt unknown /proc/32679/task/86185/exe (readlink: Permission denied) 2024-03-28T15:55:40.513 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86185 safe_time ceph NOFD /proc/32679/task/86185/fd (opendir: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86186 mgr-fin ceph cwd unknown /proc/32679/task/86186/cwd (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86186 mgr-fin ceph rtd unknown /proc/32679/task/86186/root (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86186 mgr-fin ceph txt unknown /proc/32679/task/86186/exe (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86186 mgr-fin ceph NOFD /proc/32679/task/86186/fd (opendir: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86187 mgr-fin ceph cwd unknown /proc/32679/task/86187/cwd (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86187 mgr-fin ceph rtd unknown /proc/32679/task/86187/root (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86187 mgr-fin ceph txt unknown /proc/32679/task/86187/exe (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86187 mgr-fin ceph NOFD /proc/32679/task/86187/fd (opendir: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86188 mgr-fin ceph cwd unknown /proc/32679/task/86188/cwd (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86188 mgr-fin ceph rtd unknown /proc/32679/task/86188/root (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86188 mgr-fin ceph txt unknown /proc/32679/task/86188/exe (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86188 mgr-fin ceph NOFD /proc/32679/task/86188/fd (opendir: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86189 mgr-fin ceph cwd unknown /proc/32679/task/86189/cwd (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86189 mgr-fin ceph rtd unknown /proc/32679/task/86189/root (readlink: Permission denied) 2024-03-28T15:55:40.514 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86189 mgr-fin ceph txt unknown /proc/32679/task/86189/exe (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86189 mgr-fin ceph NOFD /proc/32679/task/86189/fd (opendir: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86190 mgr-fin ceph cwd unknown /proc/32679/task/86190/cwd (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86190 mgr-fin ceph rtd unknown /proc/32679/task/86190/root (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86190 mgr-fin ceph txt unknown /proc/32679/task/86190/exe (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86190 mgr-fin ceph NOFD /proc/32679/task/86190/fd (opendir: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86191 mgr-fin ceph cwd unknown /proc/32679/task/86191/cwd (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86191 mgr-fin ceph rtd unknown /proc/32679/task/86191/root (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86191 mgr-fin ceph txt unknown /proc/32679/task/86191/exe (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86191 mgr-fin ceph NOFD /proc/32679/task/86191/fd (opendir: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86192 mgr-fin ceph cwd unknown /proc/32679/task/86192/cwd (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86192 mgr-fin ceph rtd unknown /proc/32679/task/86192/root (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86192 mgr-fin ceph txt unknown /proc/32679/task/86192/exe (readlink: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86192 mgr-fin ceph NOFD /proc/32679/task/86192/fd (opendir: Permission denied) 2024-03-28T15:55:40.515 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86193 mgr-fin ceph cwd unknown /proc/32679/task/86193/cwd (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86193 mgr-fin ceph rtd unknown /proc/32679/task/86193/root (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86193 mgr-fin ceph txt unknown /proc/32679/task/86193/exe (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86193 mgr-fin ceph NOFD /proc/32679/task/86193/fd (opendir: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86194 mgr-fin ceph cwd unknown /proc/32679/task/86194/cwd (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86194 mgr-fin ceph rtd unknown /proc/32679/task/86194/root (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86194 mgr-fin ceph txt unknown /proc/32679/task/86194/exe (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86194 mgr-fin ceph NOFD /proc/32679/task/86194/fd (opendir: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86195 mgr-fin ceph cwd unknown /proc/32679/task/86195/cwd (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86195 mgr-fin ceph rtd unknown /proc/32679/task/86195/root (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86195 mgr-fin ceph txt unknown /proc/32679/task/86195/exe (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86195 mgr-fin ceph NOFD /proc/32679/task/86195/fd (opendir: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86196 mgr-fin ceph cwd unknown /proc/32679/task/86196/cwd (readlink: Permission denied) 2024-03-28T15:55:40.516 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86196 mgr-fin ceph rtd unknown /proc/32679/task/86196/root (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86196 mgr-fin ceph txt unknown /proc/32679/task/86196/exe (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86196 mgr-fin ceph NOFD /proc/32679/task/86196/fd (opendir: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86198 mgr-fin ceph cwd unknown /proc/32679/task/86198/cwd (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86198 mgr-fin ceph rtd unknown /proc/32679/task/86198/root (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86198 mgr-fin ceph txt unknown /proc/32679/task/86198/exe (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86198 mgr-fin ceph NOFD /proc/32679/task/86198/fd (opendir: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86200 m-fin-vol ceph cwd unknown /proc/32679/task/86200/cwd (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86200 m-fin-vol ceph rtd unknown /proc/32679/task/86200/root (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86200 m-fin-vol ceph txt unknown /proc/32679/task/86200/exe (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86200 m-fin-vol ceph NOFD /proc/32679/task/86200/fd (opendir: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86201 io_contex ceph cwd unknown /proc/32679/task/86201/cwd (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86201 io_contex ceph rtd unknown /proc/32679/task/86201/root (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86201 io_contex ceph txt unknown /proc/32679/task/86201/exe (readlink: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86201 io_contex ceph NOFD /proc/32679/task/86201/fd (opendir: Permission denied) 2024-03-28T15:55:40.517 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86202 io_contex ceph cwd unknown /proc/32679/task/86202/cwd (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86202 io_contex ceph rtd unknown /proc/32679/task/86202/root (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86202 io_contex ceph txt unknown /proc/32679/task/86202/exe (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86202 io_contex ceph NOFD /proc/32679/task/86202/fd (opendir: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86236 mgr-fin ceph cwd unknown /proc/32679/task/86236/cwd (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86236 mgr-fin ceph rtd unknown /proc/32679/task/86236/root (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86236 mgr-fin ceph txt unknown /proc/32679/task/86236/exe (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86236 mgr-fin ceph NOFD /proc/32679/task/86236/fd (opendir: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86243 mgr-fin ceph cwd unknown /proc/32679/task/86243/cwd (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86243 mgr-fin ceph rtd unknown /proc/32679/task/86243/root (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86243 mgr-fin ceph txt unknown /proc/32679/task/86243/exe (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86243 mgr-fin ceph NOFD /proc/32679/task/86243/fd (opendir: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86244 mgr-fin ceph cwd unknown /proc/32679/task/86244/cwd (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86244 mgr-fin ceph rtd unknown /proc/32679/task/86244/root (readlink: Permission denied) 2024-03-28T15:55:40.518 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86244 mgr-fin ceph txt unknown /proc/32679/task/86244/exe (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86244 mgr-fin ceph NOFD /proc/32679/task/86244/fd (opendir: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86245 mgr-fin ceph cwd unknown /proc/32679/task/86245/cwd (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86245 mgr-fin ceph rtd unknown /proc/32679/task/86245/root (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86245 mgr-fin ceph txt unknown /proc/32679/task/86245/exe (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86245 mgr-fin ceph NOFD /proc/32679/task/86245/fd (opendir: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86247 mgr-fin ceph cwd unknown /proc/32679/task/86247/cwd (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86247 mgr-fin ceph rtd unknown /proc/32679/task/86247/root (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86247 mgr-fin ceph txt unknown /proc/32679/task/86247/exe (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86247 mgr-fin ceph NOFD /proc/32679/task/86247/fd (opendir: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86248 mgr-fin ceph cwd unknown /proc/32679/task/86248/cwd (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86248 mgr-fin ceph rtd unknown /proc/32679/task/86248/root (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86248 mgr-fin ceph txt unknown /proc/32679/task/86248/exe (readlink: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86248 mgr-fin ceph NOFD /proc/32679/task/86248/fd (opendir: Permission denied) 2024-03-28T15:55:40.519 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86249 mgr-fin ceph cwd unknown /proc/32679/task/86249/cwd (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86249 mgr-fin ceph rtd unknown /proc/32679/task/86249/root (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86249 mgr-fin ceph txt unknown /proc/32679/task/86249/exe (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86249 mgr-fin ceph NOFD /proc/32679/task/86249/fd (opendir: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86250 mgr-fin ceph cwd unknown /proc/32679/task/86250/cwd (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86250 mgr-fin ceph rtd unknown /proc/32679/task/86250/root (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86250 mgr-fin ceph txt unknown /proc/32679/task/86250/exe (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86250 mgr-fin ceph NOFD /proc/32679/task/86250/fd (opendir: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86251 mgr-fin ceph cwd unknown /proc/32679/task/86251/cwd (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86251 mgr-fin ceph rtd unknown /proc/32679/task/86251/root (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86251 mgr-fin ceph txt unknown /proc/32679/task/86251/exe (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86251 mgr-fin ceph NOFD /proc/32679/task/86251/fd (opendir: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86254 mgr-fin ceph cwd unknown /proc/32679/task/86254/cwd (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86254 mgr-fin ceph rtd unknown /proc/32679/task/86254/root (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86254 mgr-fin ceph txt unknown /proc/32679/task/86254/exe (readlink: Permission denied) 2024-03-28T15:55:40.520 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86254 mgr-fin ceph NOFD /proc/32679/task/86254/fd (opendir: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86255 mgr-fin ceph cwd unknown /proc/32679/task/86255/cwd (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86255 mgr-fin ceph rtd unknown /proc/32679/task/86255/root (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86255 mgr-fin ceph txt unknown /proc/32679/task/86255/exe (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86255 mgr-fin ceph NOFD /proc/32679/task/86255/fd (opendir: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86256 dashboard ceph cwd unknown /proc/32679/task/86256/cwd (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86256 dashboard ceph rtd unknown /proc/32679/task/86256/root (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86256 dashboard ceph txt unknown /proc/32679/task/86256/exe (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86256 dashboard ceph NOFD /proc/32679/task/86256/fd (opendir: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86257 dashboard ceph cwd unknown /proc/32679/task/86257/cwd (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86257 dashboard ceph rtd unknown /proc/32679/task/86257/root (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86257 dashboard ceph txt unknown /proc/32679/task/86257/exe (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86257 dashboard ceph NOFD /proc/32679/task/86257/fd (opendir: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86258 dashboard ceph cwd unknown /proc/32679/task/86258/cwd (readlink: Permission denied) 2024-03-28T15:55:40.521 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86258 dashboard ceph rtd unknown /proc/32679/task/86258/root (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86258 dashboard ceph txt unknown /proc/32679/task/86258/exe (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86258 dashboard ceph NOFD /proc/32679/task/86258/fd (opendir: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86259 dashboard ceph cwd unknown /proc/32679/task/86259/cwd (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86259 dashboard ceph rtd unknown /proc/32679/task/86259/root (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86259 dashboard ceph txt unknown /proc/32679/task/86259/exe (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86259 dashboard ceph NOFD /proc/32679/task/86259/fd (opendir: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86260 dashboard ceph cwd unknown /proc/32679/task/86260/cwd (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86260 dashboard ceph rtd unknown /proc/32679/task/86260/root (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86260 dashboard ceph txt unknown /proc/32679/task/86260/exe (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86260 dashboard ceph NOFD /proc/32679/task/86260/fd (opendir: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86261 dashboard ceph cwd unknown /proc/32679/task/86261/cwd (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86261 dashboard ceph rtd unknown /proc/32679/task/86261/root (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86261 dashboard ceph txt unknown /proc/32679/task/86261/exe (readlink: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86261 dashboard ceph NOFD /proc/32679/task/86261/fd (opendir: Permission denied) 2024-03-28T15:55:40.522 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86262 dashboard ceph cwd unknown /proc/32679/task/86262/cwd (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86262 dashboard ceph rtd unknown /proc/32679/task/86262/root (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86262 dashboard ceph txt unknown /proc/32679/task/86262/exe (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86262 dashboard ceph NOFD /proc/32679/task/86262/fd (opendir: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86263 dashboard ceph cwd unknown /proc/32679/task/86263/cwd (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86263 dashboard ceph rtd unknown /proc/32679/task/86263/root (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86263 dashboard ceph txt unknown /proc/32679/task/86263/exe (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86263 dashboard ceph NOFD /proc/32679/task/86263/fd (opendir: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86264 dashboard ceph cwd unknown /proc/32679/task/86264/cwd (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86264 dashboard ceph rtd unknown /proc/32679/task/86264/root (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86264 dashboard ceph txt unknown /proc/32679/task/86264/exe (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86264 dashboard ceph NOFD /proc/32679/task/86264/fd (opendir: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86265 dashboard ceph cwd unknown /proc/32679/task/86265/cwd (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86265 dashboard ceph rtd unknown /proc/32679/task/86265/root (readlink: Permission denied) 2024-03-28T15:55:40.523 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86265 dashboard ceph txt unknown /proc/32679/task/86265/exe (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86265 dashboard ceph NOFD /proc/32679/task/86265/fd (opendir: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86266 dashboard ceph cwd unknown /proc/32679/task/86266/cwd (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86266 dashboard ceph rtd unknown /proc/32679/task/86266/root (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86266 dashboard ceph txt unknown /proc/32679/task/86266/exe (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86266 dashboard ceph NOFD /proc/32679/task/86266/fd (opendir: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86317 dashboard ceph cwd unknown /proc/32679/task/86317/cwd (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86317 dashboard ceph rtd unknown /proc/32679/task/86317/root (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86317 dashboard ceph txt unknown /proc/32679/task/86317/exe (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86317 dashboard ceph NOFD /proc/32679/task/86317/fd (opendir: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86319 mgr-fin ceph cwd unknown /proc/32679/task/86319/cwd (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86319 mgr-fin ceph rtd unknown /proc/32679/task/86319/root (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86319 mgr-fin ceph txt unknown /proc/32679/task/86319/exe (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86319 mgr-fin ceph NOFD /proc/32679/task/86319/fd (opendir: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86320 mgr-fin ceph cwd unknown /proc/32679/task/86320/cwd (readlink: Permission denied) 2024-03-28T15:55:40.524 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86320 mgr-fin ceph rtd unknown /proc/32679/task/86320/root (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86320 mgr-fin ceph txt unknown /proc/32679/task/86320/exe (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86320 mgr-fin ceph NOFD /proc/32679/task/86320/fd (opendir: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86321 mgr-fin ceph cwd unknown /proc/32679/task/86321/cwd (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86321 mgr-fin ceph rtd unknown /proc/32679/task/86321/root (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86321 mgr-fin ceph txt unknown /proc/32679/task/86321/exe (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86321 mgr-fin ceph NOFD /proc/32679/task/86321/fd (opendir: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86322 mgr-fin ceph cwd unknown /proc/32679/task/86322/cwd (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86322 mgr-fin ceph rtd unknown /proc/32679/task/86322/root (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86322 mgr-fin ceph txt unknown /proc/32679/task/86322/exe (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86322 mgr-fin ceph NOFD /proc/32679/task/86322/fd (opendir: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86323 mgr-fin ceph cwd unknown /proc/32679/task/86323/cwd (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86323 mgr-fin ceph rtd unknown /proc/32679/task/86323/root (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86323 mgr-fin ceph txt unknown /proc/32679/task/86323/exe (readlink: Permission denied) 2024-03-28T15:55:40.525 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86323 mgr-fin ceph NOFD /proc/32679/task/86323/fd (opendir: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86324 mgr-fin ceph cwd unknown /proc/32679/task/86324/cwd (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86324 mgr-fin ceph rtd unknown /proc/32679/task/86324/root (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86324 mgr-fin ceph txt unknown /proc/32679/task/86324/exe (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86324 mgr-fin ceph NOFD /proc/32679/task/86324/fd (opendir: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86325 mgr-fin ceph cwd unknown /proc/32679/task/86325/cwd (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86325 mgr-fin ceph rtd unknown /proc/32679/task/86325/root (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86325 mgr-fin ceph txt unknown /proc/32679/task/86325/exe (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86325 mgr-fin ceph NOFD /proc/32679/task/86325/fd (opendir: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86326 mgr-fin ceph cwd unknown /proc/32679/task/86326/cwd (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86326 mgr-fin ceph rtd unknown /proc/32679/task/86326/root (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86326 mgr-fin ceph txt unknown /proc/32679/task/86326/exe (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86326 mgr-fin ceph NOFD /proc/32679/task/86326/fd (opendir: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86327 mgr-fin ceph cwd unknown /proc/32679/task/86327/cwd (readlink: Permission denied) 2024-03-28T15:55:40.526 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86327 mgr-fin ceph rtd unknown /proc/32679/task/86327/root (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86327 mgr-fin ceph txt unknown /proc/32679/task/86327/exe (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86327 mgr-fin ceph NOFD /proc/32679/task/86327/fd (opendir: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86328 mgr-fin ceph cwd unknown /proc/32679/task/86328/cwd (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86328 mgr-fin ceph rtd unknown /proc/32679/task/86328/root (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86328 mgr-fin ceph txt unknown /proc/32679/task/86328/exe (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86328 mgr-fin ceph NOFD /proc/32679/task/86328/fd (opendir: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86329 mgr-fin ceph cwd unknown /proc/32679/task/86329/cwd (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86329 mgr-fin ceph rtd unknown /proc/32679/task/86329/root (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86329 mgr-fin ceph txt unknown /proc/32679/task/86329/exe (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86329 mgr-fin ceph NOFD /proc/32679/task/86329/fd (opendir: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86672 devicehea ceph cwd unknown /proc/32679/task/86672/cwd (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86672 devicehea ceph rtd unknown /proc/32679/task/86672/root (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86672 devicehea ceph txt unknown /proc/32679/task/86672/exe (readlink: Permission denied) 2024-03-28T15:55:40.527 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86672 devicehea ceph NOFD /proc/32679/task/86672/fd (opendir: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86680 io_contex ceph cwd unknown /proc/32679/task/86680/cwd (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86680 io_contex ceph rtd unknown /proc/32679/task/86680/root (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86680 io_contex ceph txt unknown /proc/32679/task/86680/exe (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86680 io_contex ceph NOFD /proc/32679/task/86680/fd (opendir: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86681 io_contex ceph cwd unknown /proc/32679/task/86681/cwd (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86681 io_contex ceph rtd unknown /proc/32679/task/86681/root (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86681 io_contex ceph txt unknown /proc/32679/task/86681/exe (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86681 io_contex ceph NOFD /proc/32679/task/86681/fd (opendir: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86682 ceph_time ceph cwd unknown /proc/32679/task/86682/cwd (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86682 ceph_time ceph rtd unknown /proc/32679/task/86682/root (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86682 ceph_time ceph txt unknown /proc/32679/task/86682/exe (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86682 ceph_time ceph NOFD /proc/32679/task/86682/fd (opendir: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86683 ms_dispat ceph cwd unknown /proc/32679/task/86683/cwd (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86683 ms_dispat ceph rtd unknown /proc/32679/task/86683/root (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86683 ms_dispat ceph txt unknown /proc/32679/task/86683/exe (readlink: Permission denied) 2024-03-28T15:55:40.528 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86683 ms_dispat ceph NOFD /proc/32679/task/86683/fd (opendir: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86684 ms_local ceph cwd unknown /proc/32679/task/86684/cwd (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86684 ms_local ceph rtd unknown /proc/32679/task/86684/root (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86684 ms_local ceph txt unknown /proc/32679/task/86684/exe (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86684 ms_local ceph NOFD /proc/32679/task/86684/fd (opendir: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86685 safe_time ceph cwd unknown /proc/32679/task/86685/cwd (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86685 safe_time ceph rtd unknown /proc/32679/task/86685/root (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86685 safe_time ceph txt unknown /proc/32679/task/86685/exe (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86685 safe_time ceph NOFD /proc/32679/task/86685/fd (opendir: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86687 safe_time ceph cwd unknown /proc/32679/task/86687/cwd (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86687 safe_time ceph rtd unknown /proc/32679/task/86687/root (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86687 safe_time ceph txt unknown /proc/32679/task/86687/exe (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 86687 safe_time ceph NOFD /proc/32679/task/86687/fd (opendir: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88103 ceph_time ceph cwd unknown /proc/32679/task/88103/cwd (readlink: Permission denied) 2024-03-28T15:55:40.529 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88103 ceph_time ceph rtd unknown /proc/32679/task/88103/root (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88103 ceph_time ceph txt unknown /proc/32679/task/88103/exe (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88103 ceph_time ceph NOFD /proc/32679/task/88103/fd (opendir: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88104 fn_anonym ceph cwd unknown /proc/32679/task/88104/cwd (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88104 fn_anonym ceph rtd unknown /proc/32679/task/88104/root (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88104 fn_anonym ceph txt unknown /proc/32679/task/88104/exe (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88104 fn_anonym ceph NOFD /proc/32679/task/88104/fd (opendir: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88105 safe_time ceph cwd unknown /proc/32679/task/88105/cwd (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88105 safe_time ceph rtd unknown /proc/32679/task/88105/root (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88105 safe_time ceph txt unknown /proc/32679/task/88105/exe (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88105 safe_time ceph NOFD /proc/32679/task/88105/fd (opendir: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88106 fn_anonym ceph cwd unknown /proc/32679/task/88106/cwd (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88106 fn_anonym ceph rtd unknown /proc/32679/task/88106/root (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88106 fn_anonym ceph txt unknown /proc/32679/task/88106/exe (readlink: Permission denied) 2024-03-28T15:55:40.530 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88106 fn_anonym ceph NOFD /proc/32679/task/88106/fd (opendir: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88107 flusher ceph cwd unknown /proc/32679/task/88107/cwd (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88107 flusher ceph rtd unknown /proc/32679/task/88107/root (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88107 flusher ceph txt unknown /proc/32679/task/88107/exe (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88107 flusher ceph NOFD /proc/32679/task/88107/fd (opendir: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88108 ms_dispat ceph cwd unknown /proc/32679/task/88108/cwd (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88108 ms_dispat ceph rtd unknown /proc/32679/task/88108/root (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88108 ms_dispat ceph txt unknown /proc/32679/task/88108/exe (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88108 ms_dispat ceph NOFD /proc/32679/task/88108/fd (opendir: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88109 ms_local ceph cwd unknown /proc/32679/task/88109/cwd (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88109 ms_local ceph rtd unknown /proc/32679/task/88109/root (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88109 ms_local ceph txt unknown /proc/32679/task/88109/exe (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88109 ms_local ceph NOFD /proc/32679/task/88109/fd (opendir: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88110 safe_time ceph cwd unknown /proc/32679/task/88110/cwd (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88110 safe_time ceph rtd unknown /proc/32679/task/88110/root (readlink: Permission denied) 2024-03-28T15:55:40.531 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88110 safe_time ceph txt unknown /proc/32679/task/88110/exe (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88110 safe_time ceph NOFD /proc/32679/task/88110/fd (opendir: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88111 m-fin-sna ceph cwd unknown /proc/32679/task/88111/cwd (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88111 m-fin-sna ceph rtd unknown /proc/32679/task/88111/root (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88111 m-fin-sna ceph txt unknown /proc/32679/task/88111/exe (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88111 m-fin-sna ceph NOFD /proc/32679/task/88111/fd (opendir: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88657 m-fin-sna ceph cwd unknown /proc/32679/task/88657/cwd (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88657 m-fin-sna ceph rtd unknown /proc/32679/task/88657/root (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88657 m-fin-sna ceph txt unknown /proc/32679/task/88657/exe (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mgr 32679 88657 m-fin-sna ceph NOFD /proc/32679/task/88657/fd (opendir: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 33385 root cwd unknown /proc/33385/cwd (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 33385 root rtd unknown /proc/33385/root (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 33385 root txt unknown /proc/33385/exe (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 33385 root NOFD /proc/33385/fd (opendir: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 33824 root cwd unknown /proc/33824/cwd (readlink: Permission denied) 2024-03-28T15:55:40.532 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 33824 root rtd unknown /proc/33824/root (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 33824 root txt unknown /proc/33824/exe (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:systemd 33824 root NOFD /proc/33824/fd (opendir: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 33827 root cwd unknown /proc/33827/cwd (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 33827 root rtd unknown /proc/33827/root (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 33827 root txt unknown /proc/33827/exe (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:(sd-pam) 33827 root NOFD /proc/33827/fd (opendir: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 47228 root cwd unknown /proc/47228/cwd (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 47228 root rtd unknown /proc/47228/root (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 47228 root txt unknown /proc/47228/exe (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 47228 root NOFD /proc/47228/fd (opendir: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 47230 root cwd unknown /proc/47230/cwd (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 47230 root rtd unknown /proc/47230/root (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 47230 root txt unknown /proc/47230/exe (readlink: Permission denied) 2024-03-28T15:55:40.533 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 47230 root NOFD /proc/47230/fd (opendir: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 ceph cwd unknown /proc/47232/cwd (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 ceph rtd unknown /proc/47232/root (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 ceph txt unknown /proc/47232/exe (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 ceph NOFD /proc/47232/fd (opendir: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47233 log ceph cwd unknown /proc/47232/task/47233/cwd (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47233 log ceph rtd unknown /proc/47232/task/47233/root (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47233 log ceph txt unknown /proc/47232/task/47233/exe (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47233 log ceph NOFD /proc/47232/task/47233/fd (opendir: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47235 msgr-work ceph cwd unknown /proc/47232/task/47235/cwd (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47235 msgr-work ceph rtd unknown /proc/47232/task/47235/root (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47235 msgr-work ceph txt unknown /proc/47232/task/47235/exe (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47235 msgr-work ceph NOFD /proc/47232/task/47235/fd (opendir: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47236 msgr-work ceph cwd unknown /proc/47232/task/47236/cwd (readlink: Permission denied) 2024-03-28T15:55:40.534 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47236 msgr-work ceph rtd unknown /proc/47232/task/47236/root (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47236 msgr-work ceph txt unknown /proc/47232/task/47236/exe (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47236 msgr-work ceph NOFD /proc/47232/task/47236/fd (opendir: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47237 msgr-work ceph cwd unknown /proc/47232/task/47237/cwd (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47237 msgr-work ceph rtd unknown /proc/47232/task/47237/root (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47237 msgr-work ceph txt unknown /proc/47232/task/47237/exe (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47237 msgr-work ceph NOFD /proc/47232/task/47237/fd (opendir: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47241 service ceph cwd unknown /proc/47232/task/47241/cwd (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47241 service ceph rtd unknown /proc/47232/task/47241/root (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47241 service ceph txt unknown /proc/47232/task/47241/exe (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47241 service ceph NOFD /proc/47232/task/47241/fd (opendir: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47242 admin_soc ceph cwd unknown /proc/47232/task/47242/cwd (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47242 admin_soc ceph rtd unknown /proc/47232/task/47242/root (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47242 admin_soc ceph txt unknown /proc/47232/task/47242/exe (readlink: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47242 admin_soc ceph NOFD /proc/47232/task/47242/fd (opendir: Permission denied) 2024-03-28T15:55:40.535 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47384 signal_ha ceph cwd unknown /proc/47232/task/47384/cwd (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47384 signal_ha ceph rtd unknown /proc/47232/task/47384/root (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47384 signal_ha ceph txt unknown /proc/47232/task/47384/exe (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47384 signal_ha ceph NOFD /proc/47232/task/47384/fd (opendir: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47385 io_contex ceph cwd unknown /proc/47232/task/47385/cwd (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47385 io_contex ceph rtd unknown /proc/47232/task/47385/root (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47385 io_contex ceph txt unknown /proc/47232/task/47385/exe (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47385 io_contex ceph NOFD /proc/47232/task/47385/fd (opendir: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47386 io_contex ceph cwd unknown /proc/47232/task/47386/cwd (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47386 io_contex ceph rtd unknown /proc/47232/task/47386/root (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47386 io_contex ceph txt unknown /proc/47232/task/47386/exe (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47386 io_contex ceph NOFD /proc/47232/task/47386/fd (opendir: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47501 OpHistory ceph cwd unknown /proc/47232/task/47501/cwd (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47501 OpHistory ceph rtd unknown /proc/47232/task/47501/root (readlink: Permission denied) 2024-03-28T15:55:40.536 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47501 OpHistory ceph txt unknown /proc/47232/task/47501/exe (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47501 OpHistory ceph NOFD /proc/47232/task/47501/fd (opendir: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47502 ceph_time ceph cwd unknown /proc/47232/task/47502/cwd (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47502 ceph_time ceph rtd unknown /proc/47232/task/47502/root (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47502 ceph_time ceph txt unknown /proc/47232/task/47502/exe (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47502 ceph_time ceph NOFD /proc/47232/task/47502/fd (opendir: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47554 ceph-osd ceph cwd unknown /proc/47232/task/47554/cwd (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47554 ceph-osd ceph rtd unknown /proc/47232/task/47554/root (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47554 ceph-osd ceph txt unknown /proc/47232/task/47554/exe (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47554 ceph-osd ceph NOFD /proc/47232/task/47554/fd (opendir: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47559 ceph-osd ceph cwd unknown /proc/47232/task/47559/cwd (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47559 ceph-osd ceph rtd unknown /proc/47232/task/47559/root (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47559 ceph-osd ceph txt unknown /proc/47232/task/47559/exe (readlink: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47559 ceph-osd ceph NOFD /proc/47232/task/47559/fd (opendir: Permission denied) 2024-03-28T15:55:40.537 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47571 ceph-osd ceph cwd unknown /proc/47232/task/47571/cwd (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47571 ceph-osd ceph rtd unknown /proc/47232/task/47571/root (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47571 ceph-osd ceph txt unknown /proc/47232/task/47571/exe (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47571 ceph-osd ceph NOFD /proc/47232/task/47571/fd (opendir: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47590 ceph-osd ceph cwd unknown /proc/47232/task/47590/cwd (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47590 ceph-osd ceph rtd unknown /proc/47232/task/47590/root (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47590 ceph-osd ceph txt unknown /proc/47232/task/47590/exe (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47590 ceph-osd ceph NOFD /proc/47232/task/47590/fd (opendir: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47612 ceph-osd ceph cwd unknown /proc/47232/task/47612/cwd (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47612 ceph-osd ceph rtd unknown /proc/47232/task/47612/root (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47612 ceph-osd ceph txt unknown /proc/47232/task/47612/exe (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47612 ceph-osd ceph NOFD /proc/47232/task/47612/fd (opendir: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47619 ceph-osd ceph cwd unknown /proc/47232/task/47619/cwd (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47619 ceph-osd ceph rtd unknown /proc/47232/task/47619/root (readlink: Permission denied) 2024-03-28T15:55:40.538 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47619 ceph-osd ceph txt unknown /proc/47232/task/47619/exe (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47619 ceph-osd ceph NOFD /proc/47232/task/47619/fd (opendir: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47662 ceph-osd ceph cwd unknown /proc/47232/task/47662/cwd (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47662 ceph-osd ceph rtd unknown /proc/47232/task/47662/root (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47662 ceph-osd ceph txt unknown /proc/47232/task/47662/exe (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47662 ceph-osd ceph NOFD /proc/47232/task/47662/fd (opendir: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47702 ceph-osd ceph cwd unknown /proc/47232/task/47702/cwd (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47702 ceph-osd ceph rtd unknown /proc/47232/task/47702/root (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47702 ceph-osd ceph txt unknown /proc/47232/task/47702/exe (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47702 ceph-osd ceph NOFD /proc/47232/task/47702/fd (opendir: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47704 ceph-osd ceph cwd unknown /proc/47232/task/47704/cwd (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47704 ceph-osd ceph rtd unknown /proc/47232/task/47704/root (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47704 ceph-osd ceph txt unknown /proc/47232/task/47704/exe (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47704 ceph-osd ceph NOFD /proc/47232/task/47704/fd (opendir: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47705 safe_time ceph cwd unknown /proc/47232/task/47705/cwd (readlink: Permission denied) 2024-03-28T15:55:40.539 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47705 safe_time ceph rtd unknown /proc/47232/task/47705/root (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47705 safe_time ceph txt unknown /proc/47232/task/47705/exe (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47705 safe_time ceph NOFD /proc/47232/task/47705/fd (opendir: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47706 safe_time ceph cwd unknown /proc/47232/task/47706/cwd (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47706 safe_time ceph rtd unknown /proc/47232/task/47706/root (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47706 safe_time ceph txt unknown /proc/47232/task/47706/exe (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47706 safe_time ceph NOFD /proc/47232/task/47706/fd (opendir: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47707 safe_time ceph cwd unknown /proc/47232/task/47707/cwd (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47707 safe_time ceph rtd unknown /proc/47232/task/47707/root (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47707 safe_time ceph txt unknown /proc/47232/task/47707/exe (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47707 safe_time ceph NOFD /proc/47232/task/47707/fd (opendir: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47708 safe_time ceph cwd unknown /proc/47232/task/47708/cwd (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47708 safe_time ceph rtd unknown /proc/47232/task/47708/root (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47708 safe_time ceph txt unknown /proc/47232/task/47708/exe (readlink: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47708 safe_time ceph NOFD /proc/47232/task/47708/fd (opendir: Permission denied) 2024-03-28T15:55:40.540 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47709 fn_anonym ceph cwd unknown /proc/47232/task/47709/cwd (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47709 fn_anonym ceph rtd unknown /proc/47232/task/47709/root (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47709 fn_anonym ceph txt unknown /proc/47232/task/47709/exe (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47709 fn_anonym ceph NOFD /proc/47232/task/47709/fd (opendir: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47712 rocksdb:l ceph cwd unknown /proc/47232/task/47712/cwd (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47712 rocksdb:l ceph rtd unknown /proc/47232/task/47712/root (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47712 rocksdb:l ceph txt unknown /proc/47232/task/47712/exe (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47712 rocksdb:l ceph NOFD /proc/47232/task/47712/fd (opendir: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47713 rocksdb:l ceph cwd unknown /proc/47232/task/47713/cwd (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47713 rocksdb:l ceph rtd unknown /proc/47232/task/47713/root (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47713 rocksdb:l ceph txt unknown /proc/47232/task/47713/exe (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47713 rocksdb:l ceph NOFD /proc/47232/task/47713/fd (opendir: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47714 rocksdb:l ceph cwd unknown /proc/47232/task/47714/cwd (readlink: Permission denied) 2024-03-28T15:55:40.541 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47714 rocksdb:l ceph rtd unknown /proc/47232/task/47714/root (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47714 rocksdb:l ceph txt unknown /proc/47232/task/47714/exe (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47714 rocksdb:l ceph NOFD /proc/47232/task/47714/fd (opendir: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47715 rocksdb:h ceph cwd unknown /proc/47232/task/47715/cwd (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47715 rocksdb:h ceph rtd unknown /proc/47232/task/47715/root (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47715 rocksdb:h ceph txt unknown /proc/47232/task/47715/exe (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 47715 rocksdb:h ceph NOFD /proc/47232/task/47715/fd (opendir: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48120 bstore_ai ceph cwd unknown /proc/47232/task/48120/cwd (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48120 bstore_ai ceph rtd unknown /proc/47232/task/48120/root (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48120 bstore_ai ceph txt unknown /proc/47232/task/48120/exe (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48120 bstore_ai ceph NOFD /proc/47232/task/48120/fd (opendir: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48384 bstore_ai ceph cwd unknown /proc/47232/task/48384/cwd (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48384 bstore_ai ceph rtd unknown /proc/47232/task/48384/root (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48384 bstore_ai ceph txt unknown /proc/47232/task/48384/exe (readlink: Permission denied) 2024-03-28T15:55:40.542 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48384 bstore_ai ceph NOFD /proc/47232/task/48384/fd (opendir: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48566 ceph-osd ceph cwd unknown /proc/47232/task/48566/cwd (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48566 ceph-osd ceph rtd unknown /proc/47232/task/48566/root (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48566 ceph-osd ceph txt unknown /proc/47232/task/48566/exe (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48566 ceph-osd ceph NOFD /proc/47232/task/48566/fd (opendir: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48567 cfin ceph cwd unknown /proc/47232/task/48567/cwd (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48567 cfin ceph rtd unknown /proc/47232/task/48567/root (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48567 cfin ceph txt unknown /proc/47232/task/48567/exe (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48567 cfin ceph NOFD /proc/47232/task/48567/fd (opendir: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48568 bstore_kv ceph cwd unknown /proc/47232/task/48568/cwd (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48568 bstore_kv ceph rtd unknown /proc/47232/task/48568/root (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48568 bstore_kv ceph txt unknown /proc/47232/task/48568/exe (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48568 bstore_kv ceph NOFD /proc/47232/task/48568/fd (opendir: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48569 bstore_kv ceph cwd unknown /proc/47232/task/48569/cwd (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48569 bstore_kv ceph rtd unknown /proc/47232/task/48569/root (readlink: Permission denied) 2024-03-28T15:55:40.543 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48569 bstore_kv ceph txt unknown /proc/47232/task/48569/exe (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48569 bstore_kv ceph NOFD /proc/47232/task/48569/fd (opendir: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48570 bstore_me ceph cwd unknown /proc/47232/task/48570/cwd (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48570 bstore_me ceph rtd unknown /proc/47232/task/48570/root (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48570 bstore_me ceph txt unknown /proc/47232/task/48570/exe (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48570 bstore_me ceph NOFD /proc/47232/task/48570/fd (opendir: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48571 ms_dispat ceph cwd unknown /proc/47232/task/48571/cwd (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48571 ms_dispat ceph rtd unknown /proc/47232/task/48571/root (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48571 ms_dispat ceph txt unknown /proc/47232/task/48571/exe (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48571 ms_dispat ceph NOFD /proc/47232/task/48571/fd (opendir: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48572 ms_local ceph cwd unknown /proc/47232/task/48572/cwd (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48572 ms_local ceph rtd unknown /proc/47232/task/48572/root (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48572 ms_local ceph txt unknown /proc/47232/task/48572/exe (readlink: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48572 ms_local ceph NOFD /proc/47232/task/48572/fd (opendir: Permission denied) 2024-03-28T15:55:40.544 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48573 safe_time ceph cwd unknown /proc/47232/task/48573/cwd (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48573 safe_time ceph rtd unknown /proc/47232/task/48573/root (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48573 safe_time ceph txt unknown /proc/47232/task/48573/exe (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48573 safe_time ceph NOFD /proc/47232/task/48573/fd (opendir: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48574 safe_time ceph cwd unknown /proc/47232/task/48574/cwd (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48574 safe_time ceph rtd unknown /proc/47232/task/48574/root (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48574 safe_time ceph txt unknown /proc/47232/task/48574/exe (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48574 safe_time ceph NOFD /proc/47232/task/48574/fd (opendir: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48575 ms_dispat ceph cwd unknown /proc/47232/task/48575/cwd (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48575 ms_dispat ceph rtd unknown /proc/47232/task/48575/root (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48575 ms_dispat ceph txt unknown /proc/47232/task/48575/exe (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48575 ms_dispat ceph NOFD /proc/47232/task/48575/fd (opendir: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48576 ms_local ceph cwd unknown /proc/47232/task/48576/cwd (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48576 ms_local ceph rtd unknown /proc/47232/task/48576/root (readlink: Permission denied) 2024-03-28T15:55:40.545 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48576 ms_local ceph txt unknown /proc/47232/task/48576/exe (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48576 ms_local ceph NOFD /proc/47232/task/48576/fd (opendir: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48577 ms_dispat ceph cwd unknown /proc/47232/task/48577/cwd (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48577 ms_dispat ceph rtd unknown /proc/47232/task/48577/root (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48577 ms_dispat ceph txt unknown /proc/47232/task/48577/exe (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48577 ms_dispat ceph NOFD /proc/47232/task/48577/fd (opendir: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48578 ms_local ceph cwd unknown /proc/47232/task/48578/cwd (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48578 ms_local ceph rtd unknown /proc/47232/task/48578/root (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48578 ms_local ceph txt unknown /proc/47232/task/48578/exe (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48578 ms_local ceph NOFD /proc/47232/task/48578/fd (opendir: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48579 ms_dispat ceph cwd unknown /proc/47232/task/48579/cwd (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48579 ms_dispat ceph rtd unknown /proc/47232/task/48579/root (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48579 ms_dispat ceph txt unknown /proc/47232/task/48579/exe (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48579 ms_dispat ceph NOFD /proc/47232/task/48579/fd (opendir: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48580 ms_local ceph cwd unknown /proc/47232/task/48580/cwd (readlink: Permission denied) 2024-03-28T15:55:40.546 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48580 ms_local ceph rtd unknown /proc/47232/task/48580/root (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48580 ms_local ceph txt unknown /proc/47232/task/48580/exe (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48580 ms_local ceph NOFD /proc/47232/task/48580/fd (opendir: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48581 ms_dispat ceph cwd unknown /proc/47232/task/48581/cwd (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48581 ms_dispat ceph rtd unknown /proc/47232/task/48581/root (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48581 ms_dispat ceph txt unknown /proc/47232/task/48581/exe (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48581 ms_dispat ceph NOFD /proc/47232/task/48581/fd (opendir: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48582 ms_local ceph cwd unknown /proc/47232/task/48582/cwd (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48582 ms_local ceph rtd unknown /proc/47232/task/48582/root (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48582 ms_local ceph txt unknown /proc/47232/task/48582/exe (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48582 ms_local ceph NOFD /proc/47232/task/48582/fd (opendir: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48583 ms_dispat ceph cwd unknown /proc/47232/task/48583/cwd (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48583 ms_dispat ceph rtd unknown /proc/47232/task/48583/root (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48583 ms_dispat ceph txt unknown /proc/47232/task/48583/exe (readlink: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48583 ms_dispat ceph NOFD /proc/47232/task/48583/fd (opendir: Permission denied) 2024-03-28T15:55:40.547 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48584 ms_local ceph cwd unknown /proc/47232/task/48584/cwd (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48584 ms_local ceph rtd unknown /proc/47232/task/48584/root (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48584 ms_local ceph txt unknown /proc/47232/task/48584/exe (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48584 ms_local ceph NOFD /proc/47232/task/48584/fd (opendir: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48585 ms_dispat ceph cwd unknown /proc/47232/task/48585/cwd (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48585 ms_dispat ceph rtd unknown /proc/47232/task/48585/root (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48585 ms_dispat ceph txt unknown /proc/47232/task/48585/exe (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48585 ms_dispat ceph NOFD /proc/47232/task/48585/fd (opendir: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48586 ms_local ceph cwd unknown /proc/47232/task/48586/cwd (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48586 ms_local ceph rtd unknown /proc/47232/task/48586/root (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48586 ms_local ceph txt unknown /proc/47232/task/48586/exe (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48586 ms_local ceph NOFD /proc/47232/task/48586/fd (opendir: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48587 fn_anonym ceph cwd unknown /proc/47232/task/48587/cwd (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48587 fn_anonym ceph rtd unknown /proc/47232/task/48587/root (readlink: Permission denied) 2024-03-28T15:55:40.548 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48587 fn_anonym ceph txt unknown /proc/47232/task/48587/exe (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48587 fn_anonym ceph NOFD /proc/47232/task/48587/fd (opendir: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48588 finisher ceph cwd unknown /proc/47232/task/48588/cwd (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48588 finisher ceph rtd unknown /proc/47232/task/48588/root (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48588 finisher ceph txt unknown /proc/47232/task/48588/exe (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48588 finisher ceph NOFD /proc/47232/task/48588/fd (opendir: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48589 safe_time ceph cwd unknown /proc/47232/task/48589/cwd (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48589 safe_time ceph rtd unknown /proc/47232/task/48589/root (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48589 safe_time ceph txt unknown /proc/47232/task/48589/exe (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48589 safe_time ceph NOFD /proc/47232/task/48589/fd (opendir: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48590 safe_time ceph cwd unknown /proc/47232/task/48590/cwd (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48590 safe_time ceph rtd unknown /proc/47232/task/48590/root (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48590 safe_time ceph txt unknown /proc/47232/task/48590/exe (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48590 safe_time ceph NOFD /proc/47232/task/48590/fd (opendir: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48591 ceph-osd ceph cwd unknown /proc/47232/task/48591/cwd (readlink: Permission denied) 2024-03-28T15:55:40.549 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48591 ceph-osd ceph rtd unknown /proc/47232/task/48591/root (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48591 ceph-osd ceph txt unknown /proc/47232/task/48591/exe (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48591 ceph-osd ceph NOFD /proc/47232/task/48591/fd (opendir: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48592 osd_srv_a ceph cwd unknown /proc/47232/task/48592/cwd (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48592 osd_srv_a ceph rtd unknown /proc/47232/task/48592/root (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48592 osd_srv_a ceph txt unknown /proc/47232/task/48592/exe (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48592 osd_srv_a ceph NOFD /proc/47232/task/48592/fd (opendir: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48593 tp_osd_tp ceph cwd unknown /proc/47232/task/48593/cwd (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48593 tp_osd_tp ceph rtd unknown /proc/47232/task/48593/root (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48593 tp_osd_tp ceph txt unknown /proc/47232/task/48593/exe (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48593 tp_osd_tp ceph NOFD /proc/47232/task/48593/fd (opendir: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48594 tp_osd_tp ceph cwd unknown /proc/47232/task/48594/cwd (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48594 tp_osd_tp ceph rtd unknown /proc/47232/task/48594/root (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48594 tp_osd_tp ceph txt unknown /proc/47232/task/48594/exe (readlink: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48594 tp_osd_tp ceph NOFD /proc/47232/task/48594/fd (opendir: Permission denied) 2024-03-28T15:55:40.550 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48595 tp_osd_tp ceph cwd unknown /proc/47232/task/48595/cwd (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48595 tp_osd_tp ceph rtd unknown /proc/47232/task/48595/root (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48595 tp_osd_tp ceph txt unknown /proc/47232/task/48595/exe (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48595 tp_osd_tp ceph NOFD /proc/47232/task/48595/fd (opendir: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48596 tp_osd_tp ceph cwd unknown /proc/47232/task/48596/cwd (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48596 tp_osd_tp ceph rtd unknown /proc/47232/task/48596/root (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48596 tp_osd_tp ceph txt unknown /proc/47232/task/48596/exe (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48596 tp_osd_tp ceph NOFD /proc/47232/task/48596/fd (opendir: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48597 tp_osd_tp ceph cwd unknown /proc/47232/task/48597/cwd (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48597 tp_osd_tp ceph rtd unknown /proc/47232/task/48597/root (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48597 tp_osd_tp ceph txt unknown /proc/47232/task/48597/exe (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48597 tp_osd_tp ceph NOFD /proc/47232/task/48597/fd (opendir: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48598 tp_osd_tp ceph cwd unknown /proc/47232/task/48598/cwd (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48598 tp_osd_tp ceph rtd unknown /proc/47232/task/48598/root (readlink: Permission denied) 2024-03-28T15:55:40.551 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48598 tp_osd_tp ceph txt unknown /proc/47232/task/48598/exe (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48598 tp_osd_tp ceph NOFD /proc/47232/task/48598/fd (opendir: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48599 tp_osd_tp ceph cwd unknown /proc/47232/task/48599/cwd (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48599 tp_osd_tp ceph rtd unknown /proc/47232/task/48599/root (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48599 tp_osd_tp ceph txt unknown /proc/47232/task/48599/exe (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48599 tp_osd_tp ceph NOFD /proc/47232/task/48599/fd (opendir: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48600 tp_osd_tp ceph cwd unknown /proc/47232/task/48600/cwd (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48600 tp_osd_tp ceph rtd unknown /proc/47232/task/48600/root (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48600 tp_osd_tp ceph txt unknown /proc/47232/task/48600/exe (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48600 tp_osd_tp ceph NOFD /proc/47232/task/48600/fd (opendir: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48601 tp_osd_tp ceph cwd unknown /proc/47232/task/48601/cwd (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48601 tp_osd_tp ceph rtd unknown /proc/47232/task/48601/root (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48601 tp_osd_tp ceph txt unknown /proc/47232/task/48601/exe (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48601 tp_osd_tp ceph NOFD /proc/47232/task/48601/fd (opendir: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48602 tp_osd_tp ceph cwd unknown /proc/47232/task/48602/cwd (readlink: Permission denied) 2024-03-28T15:55:40.552 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48602 tp_osd_tp ceph rtd unknown /proc/47232/task/48602/root (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48602 tp_osd_tp ceph txt unknown /proc/47232/task/48602/exe (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48602 tp_osd_tp ceph NOFD /proc/47232/task/48602/fd (opendir: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48603 tp_osd_tp ceph cwd unknown /proc/47232/task/48603/cwd (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48603 tp_osd_tp ceph rtd unknown /proc/47232/task/48603/root (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48603 tp_osd_tp ceph txt unknown /proc/47232/task/48603/exe (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48603 tp_osd_tp ceph NOFD /proc/47232/task/48603/fd (opendir: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48604 tp_osd_tp ceph cwd unknown /proc/47232/task/48604/cwd (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48604 tp_osd_tp ceph rtd unknown /proc/47232/task/48604/root (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48604 tp_osd_tp ceph txt unknown /proc/47232/task/48604/exe (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48604 tp_osd_tp ceph NOFD /proc/47232/task/48604/fd (opendir: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48605 tp_osd_tp ceph cwd unknown /proc/47232/task/48605/cwd (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48605 tp_osd_tp ceph rtd unknown /proc/47232/task/48605/root (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48605 tp_osd_tp ceph txt unknown /proc/47232/task/48605/exe (readlink: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48605 tp_osd_tp ceph NOFD /proc/47232/task/48605/fd (opendir: Permission denied) 2024-03-28T15:55:40.553 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48606 tp_osd_tp ceph cwd unknown /proc/47232/task/48606/cwd (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48606 tp_osd_tp ceph rtd unknown /proc/47232/task/48606/root (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48606 tp_osd_tp ceph txt unknown /proc/47232/task/48606/exe (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48606 tp_osd_tp ceph NOFD /proc/47232/task/48606/fd (opendir: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48607 tp_osd_tp ceph cwd unknown /proc/47232/task/48607/cwd (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48607 tp_osd_tp ceph rtd unknown /proc/47232/task/48607/root (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48607 tp_osd_tp ceph txt unknown /proc/47232/task/48607/exe (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48607 tp_osd_tp ceph NOFD /proc/47232/task/48607/fd (opendir: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48608 tp_osd_tp ceph cwd unknown /proc/47232/task/48608/cwd (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48608 tp_osd_tp ceph rtd unknown /proc/47232/task/48608/root (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48608 tp_osd_tp ceph txt unknown /proc/47232/task/48608/exe (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48608 tp_osd_tp ceph NOFD /proc/47232/task/48608/fd (opendir: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48609 osd_srv_h ceph cwd unknown /proc/47232/task/48609/cwd (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48609 osd_srv_h ceph rtd unknown /proc/47232/task/48609/root (readlink: Permission denied) 2024-03-28T15:55:40.554 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48609 osd_srv_h ceph txt unknown /proc/47232/task/48609/exe (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 47232 48609 osd_srv_h ceph NOFD /proc/47232/task/48609/fd (opendir: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 48639 root cwd unknown /proc/48639/cwd (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 48639 root rtd unknown /proc/48639/root (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 48639 root txt unknown /proc/48639/exe (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 48639 root NOFD /proc/48639/fd (opendir: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 48668 root cwd unknown /proc/48668/cwd (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 48668 root rtd unknown /proc/48668/root (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 48668 root txt unknown /proc/48668/exe (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 48668 root NOFD /proc/48668/fd (opendir: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 53924 root cwd unknown /proc/53924/cwd (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 53924 root rtd unknown /proc/53924/root (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 53924 root txt unknown /proc/53924/exe (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 53924 root NOFD /proc/53924/fd (opendir: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 53926 root cwd unknown /proc/53926/cwd (readlink: Permission denied) 2024-03-28T15:55:40.555 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 53926 root rtd unknown /proc/53926/root (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 53926 root txt unknown /proc/53926/exe (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 53926 root NOFD /proc/53926/fd (opendir: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 ceph cwd unknown /proc/53928/cwd (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 ceph rtd unknown /proc/53928/root (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 ceph txt unknown /proc/53928/exe (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 ceph NOFD /proc/53928/fd (opendir: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53929 log ceph cwd unknown /proc/53928/task/53929/cwd (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53929 log ceph rtd unknown /proc/53928/task/53929/root (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53929 log ceph txt unknown /proc/53928/task/53929/exe (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53929 log ceph NOFD /proc/53928/task/53929/fd (opendir: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53931 msgr-work ceph cwd unknown /proc/53928/task/53931/cwd (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53931 msgr-work ceph rtd unknown /proc/53928/task/53931/root (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53931 msgr-work ceph txt unknown /proc/53928/task/53931/exe (readlink: Permission denied) 2024-03-28T15:55:40.556 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53931 msgr-work ceph NOFD /proc/53928/task/53931/fd (opendir: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53932 msgr-work ceph cwd unknown /proc/53928/task/53932/cwd (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53932 msgr-work ceph rtd unknown /proc/53928/task/53932/root (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53932 msgr-work ceph txt unknown /proc/53928/task/53932/exe (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53932 msgr-work ceph NOFD /proc/53928/task/53932/fd (opendir: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53933 msgr-work ceph cwd unknown /proc/53928/task/53933/cwd (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53933 msgr-work ceph rtd unknown /proc/53928/task/53933/root (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53933 msgr-work ceph txt unknown /proc/53928/task/53933/exe (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53933 msgr-work ceph NOFD /proc/53928/task/53933/fd (opendir: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53937 service ceph cwd unknown /proc/53928/task/53937/cwd (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53937 service ceph rtd unknown /proc/53928/task/53937/root (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53937 service ceph txt unknown /proc/53928/task/53937/exe (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53937 service ceph NOFD /proc/53928/task/53937/fd (opendir: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53938 admin_soc ceph cwd unknown /proc/53928/task/53938/cwd (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53938 admin_soc ceph rtd unknown /proc/53928/task/53938/root (readlink: Permission denied) 2024-03-28T15:55:40.557 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53938 admin_soc ceph txt unknown /proc/53928/task/53938/exe (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 53938 admin_soc ceph NOFD /proc/53928/task/53938/fd (opendir: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54080 signal_ha ceph cwd unknown /proc/53928/task/54080/cwd (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54080 signal_ha ceph rtd unknown /proc/53928/task/54080/root (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54080 signal_ha ceph txt unknown /proc/53928/task/54080/exe (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54080 signal_ha ceph NOFD /proc/53928/task/54080/fd (opendir: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54081 io_contex ceph cwd unknown /proc/53928/task/54081/cwd (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54081 io_contex ceph rtd unknown /proc/53928/task/54081/root (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54081 io_contex ceph txt unknown /proc/53928/task/54081/exe (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54081 io_contex ceph NOFD /proc/53928/task/54081/fd (opendir: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54082 io_contex ceph cwd unknown /proc/53928/task/54082/cwd (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54082 io_contex ceph rtd unknown /proc/53928/task/54082/root (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54082 io_contex ceph txt unknown /proc/53928/task/54082/exe (readlink: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54082 io_contex ceph NOFD /proc/53928/task/54082/fd (opendir: Permission denied) 2024-03-28T15:55:40.558 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54194 OpHistory ceph cwd unknown /proc/53928/task/54194/cwd (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54194 OpHistory ceph rtd unknown /proc/53928/task/54194/root (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54194 OpHistory ceph txt unknown /proc/53928/task/54194/exe (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54194 OpHistory ceph NOFD /proc/53928/task/54194/fd (opendir: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54195 ceph_time ceph cwd unknown /proc/53928/task/54195/cwd (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54195 ceph_time ceph rtd unknown /proc/53928/task/54195/root (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54195 ceph_time ceph txt unknown /proc/53928/task/54195/exe (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54195 ceph_time ceph NOFD /proc/53928/task/54195/fd (opendir: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54244 ceph-osd ceph cwd unknown /proc/53928/task/54244/cwd (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54244 ceph-osd ceph rtd unknown /proc/53928/task/54244/root (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54244 ceph-osd ceph txt unknown /proc/53928/task/54244/exe (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54244 ceph-osd ceph NOFD /proc/53928/task/54244/fd (opendir: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54247 ceph-osd ceph cwd unknown /proc/53928/task/54247/cwd (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54247 ceph-osd ceph rtd unknown /proc/53928/task/54247/root (readlink: Permission denied) 2024-03-28T15:55:40.559 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54247 ceph-osd ceph txt unknown /proc/53928/task/54247/exe (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54247 ceph-osd ceph NOFD /proc/53928/task/54247/fd (opendir: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54258 ceph-osd ceph cwd unknown /proc/53928/task/54258/cwd (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54258 ceph-osd ceph rtd unknown /proc/53928/task/54258/root (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54258 ceph-osd ceph txt unknown /proc/53928/task/54258/exe (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54258 ceph-osd ceph NOFD /proc/53928/task/54258/fd (opendir: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54273 ceph-osd ceph cwd unknown /proc/53928/task/54273/cwd (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54273 ceph-osd ceph rtd unknown /proc/53928/task/54273/root (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54273 ceph-osd ceph txt unknown /proc/53928/task/54273/exe (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54273 ceph-osd ceph NOFD /proc/53928/task/54273/fd (opendir: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54310 ceph-osd ceph cwd unknown /proc/53928/task/54310/cwd (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54310 ceph-osd ceph rtd unknown /proc/53928/task/54310/root (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54310 ceph-osd ceph txt unknown /proc/53928/task/54310/exe (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54310 ceph-osd ceph NOFD /proc/53928/task/54310/fd (opendir: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54316 ceph-osd ceph cwd unknown /proc/53928/task/54316/cwd (readlink: Permission denied) 2024-03-28T15:55:40.560 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54316 ceph-osd ceph rtd unknown /proc/53928/task/54316/root (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54316 ceph-osd ceph txt unknown /proc/53928/task/54316/exe (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54316 ceph-osd ceph NOFD /proc/53928/task/54316/fd (opendir: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54394 ceph-osd ceph cwd unknown /proc/53928/task/54394/cwd (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54394 ceph-osd ceph rtd unknown /proc/53928/task/54394/root (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54394 ceph-osd ceph txt unknown /proc/53928/task/54394/exe (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54394 ceph-osd ceph NOFD /proc/53928/task/54394/fd (opendir: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54405 ceph-osd ceph cwd unknown /proc/53928/task/54405/cwd (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54405 ceph-osd ceph rtd unknown /proc/53928/task/54405/root (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54405 ceph-osd ceph txt unknown /proc/53928/task/54405/exe (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54405 ceph-osd ceph NOFD /proc/53928/task/54405/fd (opendir: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54407 ceph-osd ceph cwd unknown /proc/53928/task/54407/cwd (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54407 ceph-osd ceph rtd unknown /proc/53928/task/54407/root (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54407 ceph-osd ceph txt unknown /proc/53928/task/54407/exe (readlink: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54407 ceph-osd ceph NOFD /proc/53928/task/54407/fd (opendir: Permission denied) 2024-03-28T15:55:40.561 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54408 safe_time ceph cwd unknown /proc/53928/task/54408/cwd (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54408 safe_time ceph rtd unknown /proc/53928/task/54408/root (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54408 safe_time ceph txt unknown /proc/53928/task/54408/exe (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54408 safe_time ceph NOFD /proc/53928/task/54408/fd (opendir: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54409 safe_time ceph cwd unknown /proc/53928/task/54409/cwd (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54409 safe_time ceph rtd unknown /proc/53928/task/54409/root (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54409 safe_time ceph txt unknown /proc/53928/task/54409/exe (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54409 safe_time ceph NOFD /proc/53928/task/54409/fd (opendir: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54410 safe_time ceph cwd unknown /proc/53928/task/54410/cwd (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54410 safe_time ceph rtd unknown /proc/53928/task/54410/root (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54410 safe_time ceph txt unknown /proc/53928/task/54410/exe (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54410 safe_time ceph NOFD /proc/53928/task/54410/fd (opendir: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54411 safe_time ceph cwd unknown /proc/53928/task/54411/cwd (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54411 safe_time ceph rtd unknown /proc/53928/task/54411/root (readlink: Permission denied) 2024-03-28T15:55:40.562 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54411 safe_time ceph txt unknown /proc/53928/task/54411/exe (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54411 safe_time ceph NOFD /proc/53928/task/54411/fd (opendir: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54412 fn_anonym ceph cwd unknown /proc/53928/task/54412/cwd (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54412 fn_anonym ceph rtd unknown /proc/53928/task/54412/root (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54412 fn_anonym ceph txt unknown /proc/53928/task/54412/exe (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54412 fn_anonym ceph NOFD /proc/53928/task/54412/fd (opendir: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54415 rocksdb:l ceph cwd unknown /proc/53928/task/54415/cwd (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54415 rocksdb:l ceph rtd unknown /proc/53928/task/54415/root (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54415 rocksdb:l ceph txt unknown /proc/53928/task/54415/exe (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54415 rocksdb:l ceph NOFD /proc/53928/task/54415/fd (opendir: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54416 rocksdb:l ceph cwd unknown /proc/53928/task/54416/cwd (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54416 rocksdb:l ceph rtd unknown /proc/53928/task/54416/root (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54416 rocksdb:l ceph txt unknown /proc/53928/task/54416/exe (readlink: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54416 rocksdb:l ceph NOFD /proc/53928/task/54416/fd (opendir: Permission denied) 2024-03-28T15:55:40.563 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54417 rocksdb:l ceph cwd unknown /proc/53928/task/54417/cwd (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54417 rocksdb:l ceph rtd unknown /proc/53928/task/54417/root (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54417 rocksdb:l ceph txt unknown /proc/53928/task/54417/exe (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54417 rocksdb:l ceph NOFD /proc/53928/task/54417/fd (opendir: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54418 rocksdb:h ceph cwd unknown /proc/53928/task/54418/cwd (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54418 rocksdb:h ceph rtd unknown /proc/53928/task/54418/root (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54418 rocksdb:h ceph txt unknown /proc/53928/task/54418/exe (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54418 rocksdb:h ceph NOFD /proc/53928/task/54418/fd (opendir: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54825 bstore_ai ceph cwd unknown /proc/53928/task/54825/cwd (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54825 bstore_ai ceph rtd unknown /proc/53928/task/54825/root (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54825 bstore_ai ceph txt unknown /proc/53928/task/54825/exe (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 54825 bstore_ai ceph NOFD /proc/53928/task/54825/fd (opendir: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55038 bstore_ai ceph cwd unknown /proc/53928/task/55038/cwd (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55038 bstore_ai ceph rtd unknown /proc/53928/task/55038/root (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55038 bstore_ai ceph txt unknown /proc/53928/task/55038/exe (readlink: Permission denied) 2024-03-28T15:55:40.564 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55038 bstore_ai ceph NOFD /proc/53928/task/55038/fd (opendir: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55232 ceph-osd ceph cwd unknown /proc/53928/task/55232/cwd (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55232 ceph-osd ceph rtd unknown /proc/53928/task/55232/root (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55232 ceph-osd ceph txt unknown /proc/53928/task/55232/exe (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55232 ceph-osd ceph NOFD /proc/53928/task/55232/fd (opendir: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55233 cfin ceph cwd unknown /proc/53928/task/55233/cwd (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55233 cfin ceph rtd unknown /proc/53928/task/55233/root (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55233 cfin ceph txt unknown /proc/53928/task/55233/exe (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55233 cfin ceph NOFD /proc/53928/task/55233/fd (opendir: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55234 bstore_kv ceph cwd unknown /proc/53928/task/55234/cwd (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55234 bstore_kv ceph rtd unknown /proc/53928/task/55234/root (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55234 bstore_kv ceph txt unknown /proc/53928/task/55234/exe (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55234 bstore_kv ceph NOFD /proc/53928/task/55234/fd (opendir: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55235 bstore_kv ceph cwd unknown /proc/53928/task/55235/cwd (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55235 bstore_kv ceph rtd unknown /proc/53928/task/55235/root (readlink: Permission denied) 2024-03-28T15:55:40.565 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55235 bstore_kv ceph txt unknown /proc/53928/task/55235/exe (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55235 bstore_kv ceph NOFD /proc/53928/task/55235/fd (opendir: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55236 bstore_me ceph cwd unknown /proc/53928/task/55236/cwd (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55236 bstore_me ceph rtd unknown /proc/53928/task/55236/root (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55236 bstore_me ceph txt unknown /proc/53928/task/55236/exe (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55236 bstore_me ceph NOFD /proc/53928/task/55236/fd (opendir: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55248 ms_dispat ceph cwd unknown /proc/53928/task/55248/cwd (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55248 ms_dispat ceph rtd unknown /proc/53928/task/55248/root (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55248 ms_dispat ceph txt unknown /proc/53928/task/55248/exe (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55248 ms_dispat ceph NOFD /proc/53928/task/55248/fd (opendir: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55249 ms_local ceph cwd unknown /proc/53928/task/55249/cwd (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55249 ms_local ceph rtd unknown /proc/53928/task/55249/root (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55249 ms_local ceph txt unknown /proc/53928/task/55249/exe (readlink: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55249 ms_local ceph NOFD /proc/53928/task/55249/fd (opendir: Permission denied) 2024-03-28T15:55:40.566 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55250 safe_time ceph cwd unknown /proc/53928/task/55250/cwd (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55250 safe_time ceph rtd unknown /proc/53928/task/55250/root (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55250 safe_time ceph txt unknown /proc/53928/task/55250/exe (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55250 safe_time ceph NOFD /proc/53928/task/55250/fd (opendir: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55251 safe_time ceph cwd unknown /proc/53928/task/55251/cwd (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55251 safe_time ceph rtd unknown /proc/53928/task/55251/root (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55251 safe_time ceph txt unknown /proc/53928/task/55251/exe (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55251 safe_time ceph NOFD /proc/53928/task/55251/fd (opendir: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55252 ms_dispat ceph cwd unknown /proc/53928/task/55252/cwd (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55252 ms_dispat ceph rtd unknown /proc/53928/task/55252/root (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55252 ms_dispat ceph txt unknown /proc/53928/task/55252/exe (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55252 ms_dispat ceph NOFD /proc/53928/task/55252/fd (opendir: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55253 ms_local ceph cwd unknown /proc/53928/task/55253/cwd (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55253 ms_local ceph rtd unknown /proc/53928/task/55253/root (readlink: Permission denied) 2024-03-28T15:55:40.567 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55253 ms_local ceph txt unknown /proc/53928/task/55253/exe (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55253 ms_local ceph NOFD /proc/53928/task/55253/fd (opendir: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55254 ms_dispat ceph cwd unknown /proc/53928/task/55254/cwd (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55254 ms_dispat ceph rtd unknown /proc/53928/task/55254/root (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55254 ms_dispat ceph txt unknown /proc/53928/task/55254/exe (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55254 ms_dispat ceph NOFD /proc/53928/task/55254/fd (opendir: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55255 ms_local ceph cwd unknown /proc/53928/task/55255/cwd (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55255 ms_local ceph rtd unknown /proc/53928/task/55255/root (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55255 ms_local ceph txt unknown /proc/53928/task/55255/exe (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55255 ms_local ceph NOFD /proc/53928/task/55255/fd (opendir: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55256 ms_dispat ceph cwd unknown /proc/53928/task/55256/cwd (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55256 ms_dispat ceph rtd unknown /proc/53928/task/55256/root (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55256 ms_dispat ceph txt unknown /proc/53928/task/55256/exe (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55256 ms_dispat ceph NOFD /proc/53928/task/55256/fd (opendir: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55257 ms_local ceph cwd unknown /proc/53928/task/55257/cwd (readlink: Permission denied) 2024-03-28T15:55:40.568 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55257 ms_local ceph rtd unknown /proc/53928/task/55257/root (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55257 ms_local ceph txt unknown /proc/53928/task/55257/exe (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55257 ms_local ceph NOFD /proc/53928/task/55257/fd (opendir: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55258 ms_dispat ceph cwd unknown /proc/53928/task/55258/cwd (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55258 ms_dispat ceph rtd unknown /proc/53928/task/55258/root (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55258 ms_dispat ceph txt unknown /proc/53928/task/55258/exe (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55258 ms_dispat ceph NOFD /proc/53928/task/55258/fd (opendir: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55259 ms_local ceph cwd unknown /proc/53928/task/55259/cwd (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55259 ms_local ceph rtd unknown /proc/53928/task/55259/root (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55259 ms_local ceph txt unknown /proc/53928/task/55259/exe (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55259 ms_local ceph NOFD /proc/53928/task/55259/fd (opendir: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55260 ms_dispat ceph cwd unknown /proc/53928/task/55260/cwd (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55260 ms_dispat ceph rtd unknown /proc/53928/task/55260/root (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55260 ms_dispat ceph txt unknown /proc/53928/task/55260/exe (readlink: Permission denied) 2024-03-28T15:55:40.569 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55260 ms_dispat ceph NOFD /proc/53928/task/55260/fd (opendir: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55261 ms_local ceph cwd unknown /proc/53928/task/55261/cwd (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55261 ms_local ceph rtd unknown /proc/53928/task/55261/root (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55261 ms_local ceph txt unknown /proc/53928/task/55261/exe (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55261 ms_local ceph NOFD /proc/53928/task/55261/fd (opendir: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55262 ms_dispat ceph cwd unknown /proc/53928/task/55262/cwd (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55262 ms_dispat ceph rtd unknown /proc/53928/task/55262/root (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55262 ms_dispat ceph txt unknown /proc/53928/task/55262/exe (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55262 ms_dispat ceph NOFD /proc/53928/task/55262/fd (opendir: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55264 ms_local ceph cwd unknown /proc/53928/task/55264/cwd (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55264 ms_local ceph rtd unknown /proc/53928/task/55264/root (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55264 ms_local ceph txt unknown /proc/53928/task/55264/exe (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55264 ms_local ceph NOFD /proc/53928/task/55264/fd (opendir: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55265 fn_anonym ceph cwd unknown /proc/53928/task/55265/cwd (readlink: Permission denied) 2024-03-28T15:55:40.570 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55265 fn_anonym ceph rtd unknown /proc/53928/task/55265/root (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55265 fn_anonym ceph txt unknown /proc/53928/task/55265/exe (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55265 fn_anonym ceph NOFD /proc/53928/task/55265/fd (opendir: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55266 finisher ceph cwd unknown /proc/53928/task/55266/cwd (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55266 finisher ceph rtd unknown /proc/53928/task/55266/root (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55266 finisher ceph txt unknown /proc/53928/task/55266/exe (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55266 finisher ceph NOFD /proc/53928/task/55266/fd (opendir: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55267 safe_time ceph cwd unknown /proc/53928/task/55267/cwd (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55267 safe_time ceph rtd unknown /proc/53928/task/55267/root (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55267 safe_time ceph txt unknown /proc/53928/task/55267/exe (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55267 safe_time ceph NOFD /proc/53928/task/55267/fd (opendir: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55268 safe_time ceph cwd unknown /proc/53928/task/55268/cwd (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55268 safe_time ceph rtd unknown /proc/53928/task/55268/root (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55268 safe_time ceph txt unknown /proc/53928/task/55268/exe (readlink: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55268 safe_time ceph NOFD /proc/53928/task/55268/fd (opendir: Permission denied) 2024-03-28T15:55:40.571 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55269 ceph-osd ceph cwd unknown /proc/53928/task/55269/cwd (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55269 ceph-osd ceph rtd unknown /proc/53928/task/55269/root (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55269 ceph-osd ceph txt unknown /proc/53928/task/55269/exe (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55269 ceph-osd ceph NOFD /proc/53928/task/55269/fd (opendir: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55270 osd_srv_a ceph cwd unknown /proc/53928/task/55270/cwd (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55270 osd_srv_a ceph rtd unknown /proc/53928/task/55270/root (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55270 osd_srv_a ceph txt unknown /proc/53928/task/55270/exe (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55270 osd_srv_a ceph NOFD /proc/53928/task/55270/fd (opendir: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55271 tp_osd_tp ceph cwd unknown /proc/53928/task/55271/cwd (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55271 tp_osd_tp ceph rtd unknown /proc/53928/task/55271/root (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55271 tp_osd_tp ceph txt unknown /proc/53928/task/55271/exe (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55271 tp_osd_tp ceph NOFD /proc/53928/task/55271/fd (opendir: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55272 tp_osd_tp ceph cwd unknown /proc/53928/task/55272/cwd (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55272 tp_osd_tp ceph rtd unknown /proc/53928/task/55272/root (readlink: Permission denied) 2024-03-28T15:55:40.572 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55272 tp_osd_tp ceph txt unknown /proc/53928/task/55272/exe (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55272 tp_osd_tp ceph NOFD /proc/53928/task/55272/fd (opendir: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55273 tp_osd_tp ceph cwd unknown /proc/53928/task/55273/cwd (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55273 tp_osd_tp ceph rtd unknown /proc/53928/task/55273/root (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55273 tp_osd_tp ceph txt unknown /proc/53928/task/55273/exe (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55273 tp_osd_tp ceph NOFD /proc/53928/task/55273/fd (opendir: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55274 tp_osd_tp ceph cwd unknown /proc/53928/task/55274/cwd (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55274 tp_osd_tp ceph rtd unknown /proc/53928/task/55274/root (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55274 tp_osd_tp ceph txt unknown /proc/53928/task/55274/exe (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55274 tp_osd_tp ceph NOFD /proc/53928/task/55274/fd (opendir: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55275 tp_osd_tp ceph cwd unknown /proc/53928/task/55275/cwd (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55275 tp_osd_tp ceph rtd unknown /proc/53928/task/55275/root (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55275 tp_osd_tp ceph txt unknown /proc/53928/task/55275/exe (readlink: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55275 tp_osd_tp ceph NOFD /proc/53928/task/55275/fd (opendir: Permission denied) 2024-03-28T15:55:40.573 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55276 tp_osd_tp ceph cwd unknown /proc/53928/task/55276/cwd (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55276 tp_osd_tp ceph rtd unknown /proc/53928/task/55276/root (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55276 tp_osd_tp ceph txt unknown /proc/53928/task/55276/exe (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55276 tp_osd_tp ceph NOFD /proc/53928/task/55276/fd (opendir: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55277 tp_osd_tp ceph cwd unknown /proc/53928/task/55277/cwd (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55277 tp_osd_tp ceph rtd unknown /proc/53928/task/55277/root (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55277 tp_osd_tp ceph txt unknown /proc/53928/task/55277/exe (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55277 tp_osd_tp ceph NOFD /proc/53928/task/55277/fd (opendir: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55278 tp_osd_tp ceph cwd unknown /proc/53928/task/55278/cwd (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55278 tp_osd_tp ceph rtd unknown /proc/53928/task/55278/root (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55278 tp_osd_tp ceph txt unknown /proc/53928/task/55278/exe (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55278 tp_osd_tp ceph NOFD /proc/53928/task/55278/fd (opendir: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55279 tp_osd_tp ceph cwd unknown /proc/53928/task/55279/cwd (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55279 tp_osd_tp ceph rtd unknown /proc/53928/task/55279/root (readlink: Permission denied) 2024-03-28T15:55:40.574 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55279 tp_osd_tp ceph txt unknown /proc/53928/task/55279/exe (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55279 tp_osd_tp ceph NOFD /proc/53928/task/55279/fd (opendir: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55280 tp_osd_tp ceph cwd unknown /proc/53928/task/55280/cwd (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55280 tp_osd_tp ceph rtd unknown /proc/53928/task/55280/root (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55280 tp_osd_tp ceph txt unknown /proc/53928/task/55280/exe (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55280 tp_osd_tp ceph NOFD /proc/53928/task/55280/fd (opendir: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55281 tp_osd_tp ceph cwd unknown /proc/53928/task/55281/cwd (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55281 tp_osd_tp ceph rtd unknown /proc/53928/task/55281/root (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55281 tp_osd_tp ceph txt unknown /proc/53928/task/55281/exe (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55281 tp_osd_tp ceph NOFD /proc/53928/task/55281/fd (opendir: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55282 tp_osd_tp ceph cwd unknown /proc/53928/task/55282/cwd (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55282 tp_osd_tp ceph rtd unknown /proc/53928/task/55282/root (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55282 tp_osd_tp ceph txt unknown /proc/53928/task/55282/exe (readlink: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55282 tp_osd_tp ceph NOFD /proc/53928/task/55282/fd (opendir: Permission denied) 2024-03-28T15:55:40.575 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55283 tp_osd_tp ceph cwd unknown /proc/53928/task/55283/cwd (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55283 tp_osd_tp ceph rtd unknown /proc/53928/task/55283/root (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55283 tp_osd_tp ceph txt unknown /proc/53928/task/55283/exe (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55283 tp_osd_tp ceph NOFD /proc/53928/task/55283/fd (opendir: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55284 tp_osd_tp ceph cwd unknown /proc/53928/task/55284/cwd (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55284 tp_osd_tp ceph rtd unknown /proc/53928/task/55284/root (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55284 tp_osd_tp ceph txt unknown /proc/53928/task/55284/exe (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55284 tp_osd_tp ceph NOFD /proc/53928/task/55284/fd (opendir: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55285 tp_osd_tp ceph cwd unknown /proc/53928/task/55285/cwd (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55285 tp_osd_tp ceph rtd unknown /proc/53928/task/55285/root (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55285 tp_osd_tp ceph txt unknown /proc/53928/task/55285/exe (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55285 tp_osd_tp ceph NOFD /proc/53928/task/55285/fd (opendir: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55286 tp_osd_tp ceph cwd unknown /proc/53928/task/55286/cwd (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55286 tp_osd_tp ceph rtd unknown /proc/53928/task/55286/root (readlink: Permission denied) 2024-03-28T15:55:40.576 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55286 tp_osd_tp ceph txt unknown /proc/53928/task/55286/exe (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55286 tp_osd_tp ceph NOFD /proc/53928/task/55286/fd (opendir: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55287 osd_srv_h ceph cwd unknown /proc/53928/task/55287/cwd (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55287 osd_srv_h ceph rtd unknown /proc/53928/task/55287/root (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55287 osd_srv_h ceph txt unknown /proc/53928/task/55287/exe (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 53928 55287 osd_srv_h ceph NOFD /proc/53928/task/55287/fd (opendir: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 55329 root cwd unknown /proc/55329/cwd (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 55329 root rtd unknown /proc/55329/root (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 55329 root txt unknown /proc/55329/exe (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 55329 root NOFD /proc/55329/fd (opendir: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 55346 root cwd unknown /proc/55346/cwd (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 55346 root rtd unknown /proc/55346/root (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 55346 root txt unknown /proc/55346/exe (readlink: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 55346 root NOFD /proc/55346/fd (opendir: Permission denied) 2024-03-28T15:55:40.577 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57843 root cwd unknown /proc/57843/cwd (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57843 root rtd unknown /proc/57843/root (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57843 root txt unknown /proc/57843/exe (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 57843 root NOFD /proc/57843/fd (opendir: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 60448 root cwd unknown /proc/60448/cwd (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 60448 root rtd unknown /proc/60448/root (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 60448 root txt unknown /proc/60448/exe (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 60448 root NOFD /proc/60448/fd (opendir: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 60450 root cwd unknown /proc/60450/cwd (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 60450 root rtd unknown /proc/60450/root (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 60450 root txt unknown /proc/60450/exe (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 60450 root NOFD /proc/60450/fd (opendir: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 ceph cwd unknown /proc/60453/cwd (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 ceph rtd unknown /proc/60453/root (readlink: Permission denied) 2024-03-28T15:55:40.578 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 ceph txt unknown /proc/60453/exe (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 ceph NOFD /proc/60453/fd (opendir: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60454 log ceph cwd unknown /proc/60453/task/60454/cwd (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60454 log ceph rtd unknown /proc/60453/task/60454/root (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60454 log ceph txt unknown /proc/60453/task/60454/exe (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60454 log ceph NOFD /proc/60453/task/60454/fd (opendir: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60456 msgr-work ceph cwd unknown /proc/60453/task/60456/cwd (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60456 msgr-work ceph rtd unknown /proc/60453/task/60456/root (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60456 msgr-work ceph txt unknown /proc/60453/task/60456/exe (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60456 msgr-work ceph NOFD /proc/60453/task/60456/fd (opendir: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60457 msgr-work ceph cwd unknown /proc/60453/task/60457/cwd (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60457 msgr-work ceph rtd unknown /proc/60453/task/60457/root (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60457 msgr-work ceph txt unknown /proc/60453/task/60457/exe (readlink: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60457 msgr-work ceph NOFD /proc/60453/task/60457/fd (opendir: Permission denied) 2024-03-28T15:55:40.579 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60458 msgr-work ceph cwd unknown /proc/60453/task/60458/cwd (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60458 msgr-work ceph rtd unknown /proc/60453/task/60458/root (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60458 msgr-work ceph txt unknown /proc/60453/task/60458/exe (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60458 msgr-work ceph NOFD /proc/60453/task/60458/fd (opendir: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60462 service ceph cwd unknown /proc/60453/task/60462/cwd (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60462 service ceph rtd unknown /proc/60453/task/60462/root (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60462 service ceph txt unknown /proc/60453/task/60462/exe (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60462 service ceph NOFD /proc/60453/task/60462/fd (opendir: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60463 admin_soc ceph cwd unknown /proc/60453/task/60463/cwd (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60463 admin_soc ceph rtd unknown /proc/60453/task/60463/root (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60463 admin_soc ceph txt unknown /proc/60453/task/60463/exe (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60463 admin_soc ceph NOFD /proc/60453/task/60463/fd (opendir: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60614 signal_ha ceph cwd unknown /proc/60453/task/60614/cwd (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60614 signal_ha ceph rtd unknown /proc/60453/task/60614/root (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60614 signal_ha ceph txt unknown /proc/60453/task/60614/exe (readlink: Permission denied) 2024-03-28T15:55:40.580 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60614 signal_ha ceph NOFD /proc/60453/task/60614/fd (opendir: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60615 io_contex ceph cwd unknown /proc/60453/task/60615/cwd (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60615 io_contex ceph rtd unknown /proc/60453/task/60615/root (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60615 io_contex ceph txt unknown /proc/60453/task/60615/exe (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60615 io_contex ceph NOFD /proc/60453/task/60615/fd (opendir: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60616 io_contex ceph cwd unknown /proc/60453/task/60616/cwd (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60616 io_contex ceph rtd unknown /proc/60453/task/60616/root (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60616 io_contex ceph txt unknown /proc/60453/task/60616/exe (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60616 io_contex ceph NOFD /proc/60453/task/60616/fd (opendir: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60728 OpHistory ceph cwd unknown /proc/60453/task/60728/cwd (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60728 OpHistory ceph rtd unknown /proc/60453/task/60728/root (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60728 OpHistory ceph txt unknown /proc/60453/task/60728/exe (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60728 OpHistory ceph NOFD /proc/60453/task/60728/fd (opendir: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60729 ceph_time ceph cwd unknown /proc/60453/task/60729/cwd (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60729 ceph_time ceph rtd unknown /proc/60453/task/60729/root (readlink: Permission denied) 2024-03-28T15:55:40.581 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60729 ceph_time ceph txt unknown /proc/60453/task/60729/exe (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60729 ceph_time ceph NOFD /proc/60453/task/60729/fd (opendir: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60867 ceph-osd ceph cwd unknown /proc/60453/task/60867/cwd (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60867 ceph-osd ceph rtd unknown /proc/60453/task/60867/root (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60867 ceph-osd ceph txt unknown /proc/60453/task/60867/exe (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60867 ceph-osd ceph NOFD /proc/60453/task/60867/fd (opendir: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60868 safe_time ceph cwd unknown /proc/60453/task/60868/cwd (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60868 safe_time ceph rtd unknown /proc/60453/task/60868/root (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60868 safe_time ceph txt unknown /proc/60453/task/60868/exe (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60868 safe_time ceph NOFD /proc/60453/task/60868/fd (opendir: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60869 safe_time ceph cwd unknown /proc/60453/task/60869/cwd (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60869 safe_time ceph rtd unknown /proc/60453/task/60869/root (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60869 safe_time ceph txt unknown /proc/60453/task/60869/exe (readlink: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60869 safe_time ceph NOFD /proc/60453/task/60869/fd (opendir: Permission denied) 2024-03-28T15:55:40.582 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60870 safe_time ceph cwd unknown /proc/60453/task/60870/cwd (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60870 safe_time ceph rtd unknown /proc/60453/task/60870/root (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60870 safe_time ceph txt unknown /proc/60453/task/60870/exe (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60870 safe_time ceph NOFD /proc/60453/task/60870/fd (opendir: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60871 safe_time ceph cwd unknown /proc/60453/task/60871/cwd (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60871 safe_time ceph rtd unknown /proc/60453/task/60871/root (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60871 safe_time ceph txt unknown /proc/60453/task/60871/exe (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60871 safe_time ceph NOFD /proc/60453/task/60871/fd (opendir: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60872 fn_anonym ceph cwd unknown /proc/60453/task/60872/cwd (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60872 fn_anonym ceph rtd unknown /proc/60453/task/60872/root (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60872 fn_anonym ceph txt unknown /proc/60453/task/60872/exe (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60872 fn_anonym ceph NOFD /proc/60453/task/60872/fd (opendir: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60877 rocksdb:l ceph cwd unknown /proc/60453/task/60877/cwd (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60877 rocksdb:l ceph rtd unknown /proc/60453/task/60877/root (readlink: Permission denied) 2024-03-28T15:55:40.583 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60877 rocksdb:l ceph txt unknown /proc/60453/task/60877/exe (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60877 rocksdb:l ceph NOFD /proc/60453/task/60877/fd (opendir: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60878 rocksdb:l ceph cwd unknown /proc/60453/task/60878/cwd (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60878 rocksdb:l ceph rtd unknown /proc/60453/task/60878/root (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60878 rocksdb:l ceph txt unknown /proc/60453/task/60878/exe (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60878 rocksdb:l ceph NOFD /proc/60453/task/60878/fd (opendir: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60879 rocksdb:l ceph cwd unknown /proc/60453/task/60879/cwd (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60879 rocksdb:l ceph rtd unknown /proc/60453/task/60879/root (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60879 rocksdb:l ceph txt unknown /proc/60453/task/60879/exe (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60879 rocksdb:l ceph NOFD /proc/60453/task/60879/fd (opendir: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60880 rocksdb:h ceph cwd unknown /proc/60453/task/60880/cwd (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60880 rocksdb:h ceph rtd unknown /proc/60453/task/60880/root (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60880 rocksdb:h ceph txt unknown /proc/60453/task/60880/exe (readlink: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 60880 rocksdb:h ceph NOFD /proc/60453/task/60880/fd (opendir: Permission denied) 2024-03-28T15:55:40.584 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61372 bstore_ai ceph cwd unknown /proc/60453/task/61372/cwd (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61372 bstore_ai ceph rtd unknown /proc/60453/task/61372/root (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61372 bstore_ai ceph txt unknown /proc/60453/task/61372/exe (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61372 bstore_ai ceph NOFD /proc/60453/task/61372/fd (opendir: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61555 bstore_ai ceph cwd unknown /proc/60453/task/61555/cwd (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61555 bstore_ai ceph rtd unknown /proc/60453/task/61555/root (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61555 bstore_ai ceph txt unknown /proc/60453/task/61555/exe (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61555 bstore_ai ceph NOFD /proc/60453/task/61555/fd (opendir: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61737 ceph-osd ceph cwd unknown /proc/60453/task/61737/cwd (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61737 ceph-osd ceph rtd unknown /proc/60453/task/61737/root (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61737 ceph-osd ceph txt unknown /proc/60453/task/61737/exe (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61737 ceph-osd ceph NOFD /proc/60453/task/61737/fd (opendir: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61738 cfin ceph cwd unknown /proc/60453/task/61738/cwd (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61738 cfin ceph rtd unknown /proc/60453/task/61738/root (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61738 cfin ceph txt unknown /proc/60453/task/61738/exe (readlink: Permission denied) 2024-03-28T15:55:40.585 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61738 cfin ceph NOFD /proc/60453/task/61738/fd (opendir: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61739 bstore_kv ceph cwd unknown /proc/60453/task/61739/cwd (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61739 bstore_kv ceph rtd unknown /proc/60453/task/61739/root (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61739 bstore_kv ceph txt unknown /proc/60453/task/61739/exe (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61739 bstore_kv ceph NOFD /proc/60453/task/61739/fd (opendir: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61740 bstore_kv ceph cwd unknown /proc/60453/task/61740/cwd (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61740 bstore_kv ceph rtd unknown /proc/60453/task/61740/root (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61740 bstore_kv ceph txt unknown /proc/60453/task/61740/exe (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61740 bstore_kv ceph NOFD /proc/60453/task/61740/fd (opendir: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61741 bstore_me ceph cwd unknown /proc/60453/task/61741/cwd (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61741 bstore_me ceph rtd unknown /proc/60453/task/61741/root (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61741 bstore_me ceph txt unknown /proc/60453/task/61741/exe (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61741 bstore_me ceph NOFD /proc/60453/task/61741/fd (opendir: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61753 ms_dispat ceph cwd unknown /proc/60453/task/61753/cwd (readlink: Permission denied) 2024-03-28T15:55:40.586 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61753 ms_dispat ceph rtd unknown /proc/60453/task/61753/root (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61753 ms_dispat ceph txt unknown /proc/60453/task/61753/exe (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61753 ms_dispat ceph NOFD /proc/60453/task/61753/fd (opendir: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61754 ms_local ceph cwd unknown /proc/60453/task/61754/cwd (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61754 ms_local ceph rtd unknown /proc/60453/task/61754/root (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61754 ms_local ceph txt unknown /proc/60453/task/61754/exe (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61754 ms_local ceph NOFD /proc/60453/task/61754/fd (opendir: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61755 safe_time ceph cwd unknown /proc/60453/task/61755/cwd (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61755 safe_time ceph rtd unknown /proc/60453/task/61755/root (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61755 safe_time ceph txt unknown /proc/60453/task/61755/exe (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61755 safe_time ceph NOFD /proc/60453/task/61755/fd (opendir: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61757 safe_time ceph cwd unknown /proc/60453/task/61757/cwd (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61757 safe_time ceph rtd unknown /proc/60453/task/61757/root (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61757 safe_time ceph txt unknown /proc/60453/task/61757/exe (readlink: Permission denied) 2024-03-28T15:55:40.587 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61757 safe_time ceph NOFD /proc/60453/task/61757/fd (opendir: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61758 ms_dispat ceph cwd unknown /proc/60453/task/61758/cwd (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61758 ms_dispat ceph rtd unknown /proc/60453/task/61758/root (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61758 ms_dispat ceph txt unknown /proc/60453/task/61758/exe (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61758 ms_dispat ceph NOFD /proc/60453/task/61758/fd (opendir: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61759 ms_local ceph cwd unknown /proc/60453/task/61759/cwd (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61759 ms_local ceph rtd unknown /proc/60453/task/61759/root (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61759 ms_local ceph txt unknown /proc/60453/task/61759/exe (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61759 ms_local ceph NOFD /proc/60453/task/61759/fd (opendir: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61760 ms_dispat ceph cwd unknown /proc/60453/task/61760/cwd (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61760 ms_dispat ceph rtd unknown /proc/60453/task/61760/root (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61760 ms_dispat ceph txt unknown /proc/60453/task/61760/exe (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61760 ms_dispat ceph NOFD /proc/60453/task/61760/fd (opendir: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61762 ms_local ceph cwd unknown /proc/60453/task/61762/cwd (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61762 ms_local ceph rtd unknown /proc/60453/task/61762/root (readlink: Permission denied) 2024-03-28T15:55:40.588 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61762 ms_local ceph txt unknown /proc/60453/task/61762/exe (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61762 ms_local ceph NOFD /proc/60453/task/61762/fd (opendir: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61763 ms_dispat ceph cwd unknown /proc/60453/task/61763/cwd (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61763 ms_dispat ceph rtd unknown /proc/60453/task/61763/root (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61763 ms_dispat ceph txt unknown /proc/60453/task/61763/exe (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61763 ms_dispat ceph NOFD /proc/60453/task/61763/fd (opendir: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61764 ms_local ceph cwd unknown /proc/60453/task/61764/cwd (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61764 ms_local ceph rtd unknown /proc/60453/task/61764/root (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61764 ms_local ceph txt unknown /proc/60453/task/61764/exe (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61764 ms_local ceph NOFD /proc/60453/task/61764/fd (opendir: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61765 ms_dispat ceph cwd unknown /proc/60453/task/61765/cwd (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61765 ms_dispat ceph rtd unknown /proc/60453/task/61765/root (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61765 ms_dispat ceph txt unknown /proc/60453/task/61765/exe (readlink: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61765 ms_dispat ceph NOFD /proc/60453/task/61765/fd (opendir: Permission denied) 2024-03-28T15:55:40.589 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61766 ms_local ceph cwd unknown /proc/60453/task/61766/cwd (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61766 ms_local ceph rtd unknown /proc/60453/task/61766/root (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61766 ms_local ceph txt unknown /proc/60453/task/61766/exe (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61766 ms_local ceph NOFD /proc/60453/task/61766/fd (opendir: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61767 ms_dispat ceph cwd unknown /proc/60453/task/61767/cwd (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61767 ms_dispat ceph rtd unknown /proc/60453/task/61767/root (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61767 ms_dispat ceph txt unknown /proc/60453/task/61767/exe (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61767 ms_dispat ceph NOFD /proc/60453/task/61767/fd (opendir: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61768 ms_local ceph cwd unknown /proc/60453/task/61768/cwd (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61768 ms_local ceph rtd unknown /proc/60453/task/61768/root (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61768 ms_local ceph txt unknown /proc/60453/task/61768/exe (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61768 ms_local ceph NOFD /proc/60453/task/61768/fd (opendir: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61769 ms_dispat ceph cwd unknown /proc/60453/task/61769/cwd (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61769 ms_dispat ceph rtd unknown /proc/60453/task/61769/root (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61769 ms_dispat ceph txt unknown /proc/60453/task/61769/exe (readlink: Permission denied) 2024-03-28T15:55:40.590 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61769 ms_dispat ceph NOFD /proc/60453/task/61769/fd (opendir: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61770 ms_local ceph cwd unknown /proc/60453/task/61770/cwd (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61770 ms_local ceph rtd unknown /proc/60453/task/61770/root (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61770 ms_local ceph txt unknown /proc/60453/task/61770/exe (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61770 ms_local ceph NOFD /proc/60453/task/61770/fd (opendir: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61771 fn_anonym ceph cwd unknown /proc/60453/task/61771/cwd (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61771 fn_anonym ceph rtd unknown /proc/60453/task/61771/root (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61771 fn_anonym ceph txt unknown /proc/60453/task/61771/exe (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61771 fn_anonym ceph NOFD /proc/60453/task/61771/fd (opendir: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61772 finisher ceph cwd unknown /proc/60453/task/61772/cwd (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61772 finisher ceph rtd unknown /proc/60453/task/61772/root (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61772 finisher ceph txt unknown /proc/60453/task/61772/exe (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61772 finisher ceph NOFD /proc/60453/task/61772/fd (opendir: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61773 safe_time ceph cwd unknown /proc/60453/task/61773/cwd (readlink: Permission denied) 2024-03-28T15:55:40.591 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61773 safe_time ceph rtd unknown /proc/60453/task/61773/root (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61773 safe_time ceph txt unknown /proc/60453/task/61773/exe (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61773 safe_time ceph NOFD /proc/60453/task/61773/fd (opendir: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61774 safe_time ceph cwd unknown /proc/60453/task/61774/cwd (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61774 safe_time ceph rtd unknown /proc/60453/task/61774/root (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61774 safe_time ceph txt unknown /proc/60453/task/61774/exe (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61774 safe_time ceph NOFD /proc/60453/task/61774/fd (opendir: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61775 ceph-osd ceph cwd unknown /proc/60453/task/61775/cwd (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61775 ceph-osd ceph rtd unknown /proc/60453/task/61775/root (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61775 ceph-osd ceph txt unknown /proc/60453/task/61775/exe (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61775 ceph-osd ceph NOFD /proc/60453/task/61775/fd (opendir: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61776 osd_srv_a ceph cwd unknown /proc/60453/task/61776/cwd (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61776 osd_srv_a ceph rtd unknown /proc/60453/task/61776/root (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61776 osd_srv_a ceph txt unknown /proc/60453/task/61776/exe (readlink: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61776 osd_srv_a ceph NOFD /proc/60453/task/61776/fd (opendir: Permission denied) 2024-03-28T15:55:40.592 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61777 tp_osd_tp ceph cwd unknown /proc/60453/task/61777/cwd (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61777 tp_osd_tp ceph rtd unknown /proc/60453/task/61777/root (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61777 tp_osd_tp ceph txt unknown /proc/60453/task/61777/exe (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61777 tp_osd_tp ceph NOFD /proc/60453/task/61777/fd (opendir: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61778 tp_osd_tp ceph cwd unknown /proc/60453/task/61778/cwd (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61778 tp_osd_tp ceph rtd unknown /proc/60453/task/61778/root (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61778 tp_osd_tp ceph txt unknown /proc/60453/task/61778/exe (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61778 tp_osd_tp ceph NOFD /proc/60453/task/61778/fd (opendir: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61779 tp_osd_tp ceph cwd unknown /proc/60453/task/61779/cwd (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61779 tp_osd_tp ceph rtd unknown /proc/60453/task/61779/root (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61779 tp_osd_tp ceph txt unknown /proc/60453/task/61779/exe (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61779 tp_osd_tp ceph NOFD /proc/60453/task/61779/fd (opendir: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61780 tp_osd_tp ceph cwd unknown /proc/60453/task/61780/cwd (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61780 tp_osd_tp ceph rtd unknown /proc/60453/task/61780/root (readlink: Permission denied) 2024-03-28T15:55:40.593 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61780 tp_osd_tp ceph txt unknown /proc/60453/task/61780/exe (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61780 tp_osd_tp ceph NOFD /proc/60453/task/61780/fd (opendir: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61781 tp_osd_tp ceph cwd unknown /proc/60453/task/61781/cwd (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61781 tp_osd_tp ceph rtd unknown /proc/60453/task/61781/root (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61781 tp_osd_tp ceph txt unknown /proc/60453/task/61781/exe (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61781 tp_osd_tp ceph NOFD /proc/60453/task/61781/fd (opendir: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61782 tp_osd_tp ceph cwd unknown /proc/60453/task/61782/cwd (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61782 tp_osd_tp ceph rtd unknown /proc/60453/task/61782/root (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61782 tp_osd_tp ceph txt unknown /proc/60453/task/61782/exe (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61782 tp_osd_tp ceph NOFD /proc/60453/task/61782/fd (opendir: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61783 tp_osd_tp ceph cwd unknown /proc/60453/task/61783/cwd (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61783 tp_osd_tp ceph rtd unknown /proc/60453/task/61783/root (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61783 tp_osd_tp ceph txt unknown /proc/60453/task/61783/exe (readlink: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61783 tp_osd_tp ceph NOFD /proc/60453/task/61783/fd (opendir: Permission denied) 2024-03-28T15:55:40.594 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61784 tp_osd_tp ceph cwd unknown /proc/60453/task/61784/cwd (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61784 tp_osd_tp ceph rtd unknown /proc/60453/task/61784/root (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61784 tp_osd_tp ceph txt unknown /proc/60453/task/61784/exe (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61784 tp_osd_tp ceph NOFD /proc/60453/task/61784/fd (opendir: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61786 tp_osd_tp ceph cwd unknown /proc/60453/task/61786/cwd (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61786 tp_osd_tp ceph rtd unknown /proc/60453/task/61786/root (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61786 tp_osd_tp ceph txt unknown /proc/60453/task/61786/exe (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61786 tp_osd_tp ceph NOFD /proc/60453/task/61786/fd (opendir: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61787 tp_osd_tp ceph cwd unknown /proc/60453/task/61787/cwd (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61787 tp_osd_tp ceph rtd unknown /proc/60453/task/61787/root (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61787 tp_osd_tp ceph txt unknown /proc/60453/task/61787/exe (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61787 tp_osd_tp ceph NOFD /proc/60453/task/61787/fd (opendir: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61788 tp_osd_tp ceph cwd unknown /proc/60453/task/61788/cwd (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61788 tp_osd_tp ceph rtd unknown /proc/60453/task/61788/root (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61788 tp_osd_tp ceph txt unknown /proc/60453/task/61788/exe (readlink: Permission denied) 2024-03-28T15:55:40.595 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61788 tp_osd_tp ceph NOFD /proc/60453/task/61788/fd (opendir: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61789 tp_osd_tp ceph cwd unknown /proc/60453/task/61789/cwd (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61789 tp_osd_tp ceph rtd unknown /proc/60453/task/61789/root (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61789 tp_osd_tp ceph txt unknown /proc/60453/task/61789/exe (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61789 tp_osd_tp ceph NOFD /proc/60453/task/61789/fd (opendir: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61790 tp_osd_tp ceph cwd unknown /proc/60453/task/61790/cwd (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61790 tp_osd_tp ceph rtd unknown /proc/60453/task/61790/root (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61790 tp_osd_tp ceph txt unknown /proc/60453/task/61790/exe (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61790 tp_osd_tp ceph NOFD /proc/60453/task/61790/fd (opendir: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61791 tp_osd_tp ceph cwd unknown /proc/60453/task/61791/cwd (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61791 tp_osd_tp ceph rtd unknown /proc/60453/task/61791/root (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61791 tp_osd_tp ceph txt unknown /proc/60453/task/61791/exe (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61791 tp_osd_tp ceph NOFD /proc/60453/task/61791/fd (opendir: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61792 tp_osd_tp ceph cwd unknown /proc/60453/task/61792/cwd (readlink: Permission denied) 2024-03-28T15:55:40.596 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61792 tp_osd_tp ceph rtd unknown /proc/60453/task/61792/root (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61792 tp_osd_tp ceph txt unknown /proc/60453/task/61792/exe (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61792 tp_osd_tp ceph NOFD /proc/60453/task/61792/fd (opendir: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61793 tp_osd_tp ceph cwd unknown /proc/60453/task/61793/cwd (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61793 tp_osd_tp ceph rtd unknown /proc/60453/task/61793/root (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61793 tp_osd_tp ceph txt unknown /proc/60453/task/61793/exe (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61793 tp_osd_tp ceph NOFD /proc/60453/task/61793/fd (opendir: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61794 osd_srv_h ceph cwd unknown /proc/60453/task/61794/cwd (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61794 osd_srv_h ceph rtd unknown /proc/60453/task/61794/root (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61794 osd_srv_h ceph txt unknown /proc/60453/task/61794/exe (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 60453 61794 osd_srv_h ceph NOFD /proc/60453/task/61794/fd (opendir: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 61873 root cwd unknown /proc/61873/cwd (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 61873 root rtd unknown /proc/61873/root (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 61873 root txt unknown /proc/61873/exe (readlink: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 61873 root NOFD /proc/61873/fd (opendir: Permission denied) 2024-03-28T15:55:40.597 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 61890 root cwd unknown /proc/61890/cwd (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 61890 root rtd unknown /proc/61890/root (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 61890 root txt unknown /proc/61890/exe (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 61890 root NOFD /proc/61890/fd (opendir: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 64738 root cwd unknown /proc/64738/cwd (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 64738 root rtd unknown /proc/64738/root (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 64738 root txt unknown /proc/64738/exe (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 64738 root NOFD /proc/64738/fd (opendir: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66004 root cwd unknown /proc/66004/cwd (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66004 root rtd unknown /proc/66004/root (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66004 root txt unknown /proc/66004/exe (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66004 root NOFD /proc/66004/fd (opendir: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66005 root cwd unknown /proc/66005/cwd (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66005 root rtd unknown /proc/66005/root (readlink: Permission denied) 2024-03-28T15:55:40.598 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66005 root txt unknown /proc/66005/exe (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/1 66005 root NOFD /proc/66005/fd (opendir: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 67037 root cwd unknown /proc/67037/cwd (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 67037 root rtd unknown /proc/67037/root (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 67037 root txt unknown /proc/67037/exe (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 67037 root NOFD /proc/67037/fd (opendir: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 67039 root cwd unknown /proc/67039/cwd (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 67039 root rtd unknown /proc/67039/root (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 67039 root txt unknown /proc/67039/exe (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 67039 root NOFD /proc/67039/fd (opendir: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 ceph cwd unknown /proc/67041/cwd (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 ceph rtd unknown /proc/67041/root (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 ceph txt unknown /proc/67041/exe (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 ceph NOFD /proc/67041/fd (opendir: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67043 log ceph cwd unknown /proc/67041/task/67043/cwd (readlink: Permission denied) 2024-03-28T15:55:40.599 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67043 log ceph rtd unknown /proc/67041/task/67043/root (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67043 log ceph txt unknown /proc/67041/task/67043/exe (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67043 log ceph NOFD /proc/67041/task/67043/fd (opendir: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67045 msgr-work ceph cwd unknown /proc/67041/task/67045/cwd (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67045 msgr-work ceph rtd unknown /proc/67041/task/67045/root (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67045 msgr-work ceph txt unknown /proc/67041/task/67045/exe (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67045 msgr-work ceph NOFD /proc/67041/task/67045/fd (opendir: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67046 msgr-work ceph cwd unknown /proc/67041/task/67046/cwd (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67046 msgr-work ceph rtd unknown /proc/67041/task/67046/root (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67046 msgr-work ceph txt unknown /proc/67041/task/67046/exe (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67046 msgr-work ceph NOFD /proc/67041/task/67046/fd (opendir: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67047 msgr-work ceph cwd unknown /proc/67041/task/67047/cwd (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67047 msgr-work ceph rtd unknown /proc/67041/task/67047/root (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67047 msgr-work ceph txt unknown /proc/67041/task/67047/exe (readlink: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67047 msgr-work ceph NOFD /proc/67041/task/67047/fd (opendir: Permission denied) 2024-03-28T15:55:40.600 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67051 service ceph cwd unknown /proc/67041/task/67051/cwd (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67051 service ceph rtd unknown /proc/67041/task/67051/root (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67051 service ceph txt unknown /proc/67041/task/67051/exe (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67051 service ceph NOFD /proc/67041/task/67051/fd (opendir: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67052 admin_soc ceph cwd unknown /proc/67041/task/67052/cwd (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67052 admin_soc ceph rtd unknown /proc/67041/task/67052/root (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67052 admin_soc ceph txt unknown /proc/67041/task/67052/exe (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67052 admin_soc ceph NOFD /proc/67041/task/67052/fd (opendir: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67194 signal_ha ceph cwd unknown /proc/67041/task/67194/cwd (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67194 signal_ha ceph rtd unknown /proc/67041/task/67194/root (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67194 signal_ha ceph txt unknown /proc/67041/task/67194/exe (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67194 signal_ha ceph NOFD /proc/67041/task/67194/fd (opendir: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67195 io_contex ceph cwd unknown /proc/67041/task/67195/cwd (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67195 io_contex ceph rtd unknown /proc/67041/task/67195/root (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67195 io_contex ceph txt unknown /proc/67041/task/67195/exe (readlink: Permission denied) 2024-03-28T15:55:40.601 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67195 io_contex ceph NOFD /proc/67041/task/67195/fd (opendir: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67196 io_contex ceph cwd unknown /proc/67041/task/67196/cwd (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67196 io_contex ceph rtd unknown /proc/67041/task/67196/root (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67196 io_contex ceph txt unknown /proc/67041/task/67196/exe (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67196 io_contex ceph NOFD /proc/67041/task/67196/fd (opendir: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67308 OpHistory ceph cwd unknown /proc/67041/task/67308/cwd (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67308 OpHistory ceph rtd unknown /proc/67041/task/67308/root (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67308 OpHistory ceph txt unknown /proc/67041/task/67308/exe (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67308 OpHistory ceph NOFD /proc/67041/task/67308/fd (opendir: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67309 ceph_time ceph cwd unknown /proc/67041/task/67309/cwd (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67309 ceph_time ceph rtd unknown /proc/67041/task/67309/root (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67309 ceph_time ceph txt unknown /proc/67041/task/67309/exe (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67309 ceph_time ceph NOFD /proc/67041/task/67309/fd (opendir: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67479 ceph-osd ceph cwd unknown /proc/67041/task/67479/cwd (readlink: Permission denied) 2024-03-28T15:55:40.602 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67479 ceph-osd ceph rtd unknown /proc/67041/task/67479/root (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67479 ceph-osd ceph txt unknown /proc/67041/task/67479/exe (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67479 ceph-osd ceph NOFD /proc/67041/task/67479/fd (opendir: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67480 safe_time ceph cwd unknown /proc/67041/task/67480/cwd (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67480 safe_time ceph rtd unknown /proc/67041/task/67480/root (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67480 safe_time ceph txt unknown /proc/67041/task/67480/exe (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67480 safe_time ceph NOFD /proc/67041/task/67480/fd (opendir: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67482 safe_time ceph cwd unknown /proc/67041/task/67482/cwd (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67482 safe_time ceph rtd unknown /proc/67041/task/67482/root (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67482 safe_time ceph txt unknown /proc/67041/task/67482/exe (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67482 safe_time ceph NOFD /proc/67041/task/67482/fd (opendir: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67483 safe_time ceph cwd unknown /proc/67041/task/67483/cwd (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67483 safe_time ceph rtd unknown /proc/67041/task/67483/root (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67483 safe_time ceph txt unknown /proc/67041/task/67483/exe (readlink: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67483 safe_time ceph NOFD /proc/67041/task/67483/fd (opendir: Permission denied) 2024-03-28T15:55:40.603 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67484 safe_time ceph cwd unknown /proc/67041/task/67484/cwd (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67484 safe_time ceph rtd unknown /proc/67041/task/67484/root (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67484 safe_time ceph txt unknown /proc/67041/task/67484/exe (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67484 safe_time ceph NOFD /proc/67041/task/67484/fd (opendir: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67485 fn_anonym ceph cwd unknown /proc/67041/task/67485/cwd (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67485 fn_anonym ceph rtd unknown /proc/67041/task/67485/root (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67485 fn_anonym ceph txt unknown /proc/67041/task/67485/exe (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67485 fn_anonym ceph NOFD /proc/67041/task/67485/fd (opendir: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67488 rocksdb:l ceph cwd unknown /proc/67041/task/67488/cwd (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67488 rocksdb:l ceph rtd unknown /proc/67041/task/67488/root (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67488 rocksdb:l ceph txt unknown /proc/67041/task/67488/exe (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67488 rocksdb:l ceph NOFD /proc/67041/task/67488/fd (opendir: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67489 rocksdb:l ceph cwd unknown /proc/67041/task/67489/cwd (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67489 rocksdb:l ceph rtd unknown /proc/67041/task/67489/root (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67489 rocksdb:l ceph txt unknown /proc/67041/task/67489/exe (readlink: Permission denied) 2024-03-28T15:55:40.604 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67489 rocksdb:l ceph NOFD /proc/67041/task/67489/fd (opendir: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67490 rocksdb:l ceph cwd unknown /proc/67041/task/67490/cwd (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67490 rocksdb:l ceph rtd unknown /proc/67041/task/67490/root (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67490 rocksdb:l ceph txt unknown /proc/67041/task/67490/exe (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67490 rocksdb:l ceph NOFD /proc/67041/task/67490/fd (opendir: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67491 rocksdb:h ceph cwd unknown /proc/67041/task/67491/cwd (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67491 rocksdb:h ceph rtd unknown /proc/67041/task/67491/root (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67491 rocksdb:h ceph txt unknown /proc/67041/task/67491/exe (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67491 rocksdb:h ceph NOFD /proc/67041/task/67491/fd (opendir: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67970 bstore_ai ceph cwd unknown /proc/67041/task/67970/cwd (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67970 bstore_ai ceph rtd unknown /proc/67041/task/67970/root (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67970 bstore_ai ceph txt unknown /proc/67041/task/67970/exe (readlink: Permission denied) 2024-03-28T15:55:40.605 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 67970 bstore_ai ceph NOFD /proc/67041/task/67970/fd (opendir: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68152 bstore_ai ceph cwd unknown /proc/67041/task/68152/cwd (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68152 bstore_ai ceph rtd unknown /proc/67041/task/68152/root (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68152 bstore_ai ceph txt unknown /proc/67041/task/68152/exe (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68152 bstore_ai ceph NOFD /proc/67041/task/68152/fd (opendir: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68334 ceph-osd ceph cwd unknown /proc/67041/task/68334/cwd (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68334 ceph-osd ceph rtd unknown /proc/67041/task/68334/root (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68334 ceph-osd ceph txt unknown /proc/67041/task/68334/exe (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68334 ceph-osd ceph NOFD /proc/67041/task/68334/fd (opendir: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68335 cfin ceph cwd unknown /proc/67041/task/68335/cwd (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68335 cfin ceph rtd unknown /proc/67041/task/68335/root (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68335 cfin ceph txt unknown /proc/67041/task/68335/exe (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68335 cfin ceph NOFD /proc/67041/task/68335/fd (opendir: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68336 bstore_kv ceph cwd unknown /proc/67041/task/68336/cwd (readlink: Permission denied) 2024-03-28T15:55:40.606 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68336 bstore_kv ceph rtd unknown /proc/67041/task/68336/root (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68336 bstore_kv ceph txt unknown /proc/67041/task/68336/exe (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68336 bstore_kv ceph NOFD /proc/67041/task/68336/fd (opendir: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68337 bstore_kv ceph cwd unknown /proc/67041/task/68337/cwd (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68337 bstore_kv ceph rtd unknown /proc/67041/task/68337/root (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68337 bstore_kv ceph txt unknown /proc/67041/task/68337/exe (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68337 bstore_kv ceph NOFD /proc/67041/task/68337/fd (opendir: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68338 bstore_me ceph cwd unknown /proc/67041/task/68338/cwd (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68338 bstore_me ceph rtd unknown /proc/67041/task/68338/root (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68338 bstore_me ceph txt unknown /proc/67041/task/68338/exe (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68338 bstore_me ceph NOFD /proc/67041/task/68338/fd (opendir: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68339 ms_dispat ceph cwd unknown /proc/67041/task/68339/cwd (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68339 ms_dispat ceph rtd unknown /proc/67041/task/68339/root (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68339 ms_dispat ceph txt unknown /proc/67041/task/68339/exe (readlink: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68339 ms_dispat ceph NOFD /proc/67041/task/68339/fd (opendir: Permission denied) 2024-03-28T15:55:40.607 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68340 ms_local ceph cwd unknown /proc/67041/task/68340/cwd (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68340 ms_local ceph rtd unknown /proc/67041/task/68340/root (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68340 ms_local ceph txt unknown /proc/67041/task/68340/exe (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68340 ms_local ceph NOFD /proc/67041/task/68340/fd (opendir: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68341 safe_time ceph cwd unknown /proc/67041/task/68341/cwd (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68341 safe_time ceph rtd unknown /proc/67041/task/68341/root (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68341 safe_time ceph txt unknown /proc/67041/task/68341/exe (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68341 safe_time ceph NOFD /proc/67041/task/68341/fd (opendir: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68342 safe_time ceph cwd unknown /proc/67041/task/68342/cwd (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68342 safe_time ceph rtd unknown /proc/67041/task/68342/root (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68342 safe_time ceph txt unknown /proc/67041/task/68342/exe (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68342 safe_time ceph NOFD /proc/67041/task/68342/fd (opendir: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68343 ms_dispat ceph cwd unknown /proc/67041/task/68343/cwd (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68343 ms_dispat ceph rtd unknown /proc/67041/task/68343/root (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68343 ms_dispat ceph txt unknown /proc/67041/task/68343/exe (readlink: Permission denied) 2024-03-28T15:55:40.608 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68343 ms_dispat ceph NOFD /proc/67041/task/68343/fd (opendir: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68344 ms_local ceph cwd unknown /proc/67041/task/68344/cwd (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68344 ms_local ceph rtd unknown /proc/67041/task/68344/root (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68344 ms_local ceph txt unknown /proc/67041/task/68344/exe (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68344 ms_local ceph NOFD /proc/67041/task/68344/fd (opendir: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68345 ms_dispat ceph cwd unknown /proc/67041/task/68345/cwd (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68345 ms_dispat ceph rtd unknown /proc/67041/task/68345/root (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68345 ms_dispat ceph txt unknown /proc/67041/task/68345/exe (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68345 ms_dispat ceph NOFD /proc/67041/task/68345/fd (opendir: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68346 ms_local ceph cwd unknown /proc/67041/task/68346/cwd (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68346 ms_local ceph rtd unknown /proc/67041/task/68346/root (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68346 ms_local ceph txt unknown /proc/67041/task/68346/exe (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68346 ms_local ceph NOFD /proc/67041/task/68346/fd (opendir: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68347 ms_dispat ceph cwd unknown /proc/67041/task/68347/cwd (readlink: Permission denied) 2024-03-28T15:55:40.609 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68347 ms_dispat ceph rtd unknown /proc/67041/task/68347/root (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68347 ms_dispat ceph txt unknown /proc/67041/task/68347/exe (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68347 ms_dispat ceph NOFD /proc/67041/task/68347/fd (opendir: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68348 ms_local ceph cwd unknown /proc/67041/task/68348/cwd (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68348 ms_local ceph rtd unknown /proc/67041/task/68348/root (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68348 ms_local ceph txt unknown /proc/67041/task/68348/exe (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68348 ms_local ceph NOFD /proc/67041/task/68348/fd (opendir: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68349 ms_dispat ceph cwd unknown /proc/67041/task/68349/cwd (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68349 ms_dispat ceph rtd unknown /proc/67041/task/68349/root (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68349 ms_dispat ceph txt unknown /proc/67041/task/68349/exe (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68349 ms_dispat ceph NOFD /proc/67041/task/68349/fd (opendir: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68350 ms_local ceph cwd unknown /proc/67041/task/68350/cwd (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68350 ms_local ceph rtd unknown /proc/67041/task/68350/root (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68350 ms_local ceph txt unknown /proc/67041/task/68350/exe (readlink: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68350 ms_local ceph NOFD /proc/67041/task/68350/fd (opendir: Permission denied) 2024-03-28T15:55:40.610 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68351 ms_dispat ceph cwd unknown /proc/67041/task/68351/cwd (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68351 ms_dispat ceph rtd unknown /proc/67041/task/68351/root (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68351 ms_dispat ceph txt unknown /proc/67041/task/68351/exe (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68351 ms_dispat ceph NOFD /proc/67041/task/68351/fd (opendir: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68352 ms_local ceph cwd unknown /proc/67041/task/68352/cwd (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68352 ms_local ceph rtd unknown /proc/67041/task/68352/root (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68352 ms_local ceph txt unknown /proc/67041/task/68352/exe (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68352 ms_local ceph NOFD /proc/67041/task/68352/fd (opendir: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68353 ms_dispat ceph cwd unknown /proc/67041/task/68353/cwd (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68353 ms_dispat ceph rtd unknown /proc/67041/task/68353/root (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68353 ms_dispat ceph txt unknown /proc/67041/task/68353/exe (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68353 ms_dispat ceph NOFD /proc/67041/task/68353/fd (opendir: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68354 ms_local ceph cwd unknown /proc/67041/task/68354/cwd (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68354 ms_local ceph rtd unknown /proc/67041/task/68354/root (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68354 ms_local ceph txt unknown /proc/67041/task/68354/exe (readlink: Permission denied) 2024-03-28T15:55:40.611 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68354 ms_local ceph NOFD /proc/67041/task/68354/fd (opendir: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68355 fn_anonym ceph cwd unknown /proc/67041/task/68355/cwd (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68355 fn_anonym ceph rtd unknown /proc/67041/task/68355/root (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68355 fn_anonym ceph txt unknown /proc/67041/task/68355/exe (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68355 fn_anonym ceph NOFD /proc/67041/task/68355/fd (opendir: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68356 finisher ceph cwd unknown /proc/67041/task/68356/cwd (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68356 finisher ceph rtd unknown /proc/67041/task/68356/root (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68356 finisher ceph txt unknown /proc/67041/task/68356/exe (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68356 finisher ceph NOFD /proc/67041/task/68356/fd (opendir: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68357 safe_time ceph cwd unknown /proc/67041/task/68357/cwd (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68357 safe_time ceph rtd unknown /proc/67041/task/68357/root (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68357 safe_time ceph txt unknown /proc/67041/task/68357/exe (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68357 safe_time ceph NOFD /proc/67041/task/68357/fd (opendir: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68358 safe_time ceph cwd unknown /proc/67041/task/68358/cwd (readlink: Permission denied) 2024-03-28T15:55:40.612 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68358 safe_time ceph rtd unknown /proc/67041/task/68358/root (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68358 safe_time ceph txt unknown /proc/67041/task/68358/exe (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68358 safe_time ceph NOFD /proc/67041/task/68358/fd (opendir: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68359 ceph-osd ceph cwd unknown /proc/67041/task/68359/cwd (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68359 ceph-osd ceph rtd unknown /proc/67041/task/68359/root (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68359 ceph-osd ceph txt unknown /proc/67041/task/68359/exe (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68359 ceph-osd ceph NOFD /proc/67041/task/68359/fd (opendir: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68360 osd_srv_a ceph cwd unknown /proc/67041/task/68360/cwd (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68360 osd_srv_a ceph rtd unknown /proc/67041/task/68360/root (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68360 osd_srv_a ceph txt unknown /proc/67041/task/68360/exe (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68360 osd_srv_a ceph NOFD /proc/67041/task/68360/fd (opendir: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68361 tp_osd_tp ceph cwd unknown /proc/67041/task/68361/cwd (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68361 tp_osd_tp ceph rtd unknown /proc/67041/task/68361/root (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68361 tp_osd_tp ceph txt unknown /proc/67041/task/68361/exe (readlink: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68361 tp_osd_tp ceph NOFD /proc/67041/task/68361/fd (opendir: Permission denied) 2024-03-28T15:55:40.613 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68362 tp_osd_tp ceph cwd unknown /proc/67041/task/68362/cwd (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68362 tp_osd_tp ceph rtd unknown /proc/67041/task/68362/root (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68362 tp_osd_tp ceph txt unknown /proc/67041/task/68362/exe (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68362 tp_osd_tp ceph NOFD /proc/67041/task/68362/fd (opendir: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68363 tp_osd_tp ceph cwd unknown /proc/67041/task/68363/cwd (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68363 tp_osd_tp ceph rtd unknown /proc/67041/task/68363/root (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68363 tp_osd_tp ceph txt unknown /proc/67041/task/68363/exe (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68363 tp_osd_tp ceph NOFD /proc/67041/task/68363/fd (opendir: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68364 tp_osd_tp ceph cwd unknown /proc/67041/task/68364/cwd (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68364 tp_osd_tp ceph rtd unknown /proc/67041/task/68364/root (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68364 tp_osd_tp ceph txt unknown /proc/67041/task/68364/exe (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68364 tp_osd_tp ceph NOFD /proc/67041/task/68364/fd (opendir: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68365 tp_osd_tp ceph cwd unknown /proc/67041/task/68365/cwd (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68365 tp_osd_tp ceph rtd unknown /proc/67041/task/68365/root (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68365 tp_osd_tp ceph txt unknown /proc/67041/task/68365/exe (readlink: Permission denied) 2024-03-28T15:55:40.614 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68365 tp_osd_tp ceph NOFD /proc/67041/task/68365/fd (opendir: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68366 tp_osd_tp ceph cwd unknown /proc/67041/task/68366/cwd (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68366 tp_osd_tp ceph rtd unknown /proc/67041/task/68366/root (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68366 tp_osd_tp ceph txt unknown /proc/67041/task/68366/exe (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68366 tp_osd_tp ceph NOFD /proc/67041/task/68366/fd (opendir: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68367 tp_osd_tp ceph cwd unknown /proc/67041/task/68367/cwd (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68367 tp_osd_tp ceph rtd unknown /proc/67041/task/68367/root (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68367 tp_osd_tp ceph txt unknown /proc/67041/task/68367/exe (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68367 tp_osd_tp ceph NOFD /proc/67041/task/68367/fd (opendir: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68368 tp_osd_tp ceph cwd unknown /proc/67041/task/68368/cwd (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68368 tp_osd_tp ceph rtd unknown /proc/67041/task/68368/root (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68368 tp_osd_tp ceph txt unknown /proc/67041/task/68368/exe (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68368 tp_osd_tp ceph NOFD /proc/67041/task/68368/fd (opendir: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68369 tp_osd_tp ceph cwd unknown /proc/67041/task/68369/cwd (readlink: Permission denied) 2024-03-28T15:55:40.615 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68369 tp_osd_tp ceph rtd unknown /proc/67041/task/68369/root (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68369 tp_osd_tp ceph txt unknown /proc/67041/task/68369/exe (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68369 tp_osd_tp ceph NOFD /proc/67041/task/68369/fd (opendir: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68370 tp_osd_tp ceph cwd unknown /proc/67041/task/68370/cwd (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68370 tp_osd_tp ceph rtd unknown /proc/67041/task/68370/root (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68370 tp_osd_tp ceph txt unknown /proc/67041/task/68370/exe (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68370 tp_osd_tp ceph NOFD /proc/67041/task/68370/fd (opendir: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68371 tp_osd_tp ceph cwd unknown /proc/67041/task/68371/cwd (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68371 tp_osd_tp ceph rtd unknown /proc/67041/task/68371/root (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68371 tp_osd_tp ceph txt unknown /proc/67041/task/68371/exe (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68371 tp_osd_tp ceph NOFD /proc/67041/task/68371/fd (opendir: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68372 tp_osd_tp ceph cwd unknown /proc/67041/task/68372/cwd (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68372 tp_osd_tp ceph rtd unknown /proc/67041/task/68372/root (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68372 tp_osd_tp ceph txt unknown /proc/67041/task/68372/exe (readlink: Permission denied) 2024-03-28T15:55:40.616 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68372 tp_osd_tp ceph NOFD /proc/67041/task/68372/fd (opendir: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68373 tp_osd_tp ceph cwd unknown /proc/67041/task/68373/cwd (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68373 tp_osd_tp ceph rtd unknown /proc/67041/task/68373/root (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68373 tp_osd_tp ceph txt unknown /proc/67041/task/68373/exe (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68373 tp_osd_tp ceph NOFD /proc/67041/task/68373/fd (opendir: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68374 tp_osd_tp ceph cwd unknown /proc/67041/task/68374/cwd (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68374 tp_osd_tp ceph rtd unknown /proc/67041/task/68374/root (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68374 tp_osd_tp ceph txt unknown /proc/67041/task/68374/exe (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68374 tp_osd_tp ceph NOFD /proc/67041/task/68374/fd (opendir: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68375 tp_osd_tp ceph cwd unknown /proc/67041/task/68375/cwd (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68375 tp_osd_tp ceph rtd unknown /proc/67041/task/68375/root (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68375 tp_osd_tp ceph txt unknown /proc/67041/task/68375/exe (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68375 tp_osd_tp ceph NOFD /proc/67041/task/68375/fd (opendir: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68376 tp_osd_tp ceph cwd unknown /proc/67041/task/68376/cwd (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68376 tp_osd_tp ceph rtd unknown /proc/67041/task/68376/root (readlink: Permission denied) 2024-03-28T15:55:40.617 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68376 tp_osd_tp ceph txt unknown /proc/67041/task/68376/exe (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68376 tp_osd_tp ceph NOFD /proc/67041/task/68376/fd (opendir: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68377 osd_srv_h ceph cwd unknown /proc/67041/task/68377/cwd (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68377 osd_srv_h ceph rtd unknown /proc/67041/task/68377/root (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68377 osd_srv_h ceph txt unknown /proc/67041/task/68377/exe (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-osd 67041 68377 osd_srv_h ceph NOFD /proc/67041/task/68377/fd (opendir: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 68425 root cwd unknown /proc/68425/cwd (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 68425 root rtd unknown /proc/68425/root (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 68425 root txt unknown /proc/68425/exe (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 68425 root NOFD /proc/68425/fd (opendir: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 68472 root cwd unknown /proc/68472/cwd (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 68472 root rtd unknown /proc/68472/root (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 68472 root txt unknown /proc/68472/exe (readlink: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 68472 root NOFD /proc/68472/fd (opendir: Permission denied) 2024-03-28T15:55:40.618 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 68489 root cwd unknown /proc/68489/cwd (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 68489 root rtd unknown /proc/68489/root (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 68489 root txt unknown /proc/68489/exe (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 68489 root NOFD /proc/68489/fd (opendir: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 70930 root cwd unknown /proc/70930/cwd (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 70930 root rtd unknown /proc/70930/root (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 70930 root txt unknown /proc/70930/exe (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 70930 root NOFD /proc/70930/fd (opendir: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 70931 root cwd unknown /proc/70931/cwd (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 70931 root rtd unknown /proc/70931/root (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 70931 root txt unknown /proc/70931/exe (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/2 70931 root NOFD /proc/70931/fd (opendir: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71635 root cwd unknown /proc/71635/cwd (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71635 root rtd unknown /proc/71635/root (readlink: Permission denied) 2024-03-28T15:55:40.619 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71635 root txt unknown /proc/71635/exe (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71635 root NOFD /proc/71635/fd (opendir: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71652 root cwd unknown /proc/71652/cwd (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71652 root rtd unknown /proc/71652/root (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71652 root txt unknown /proc/71652/exe (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71652 root NOFD /proc/71652/fd (opendir: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71653 root cwd unknown /proc/71653/cwd (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71653 root rtd unknown /proc/71653/root (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71653 root txt unknown /proc/71653/exe (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71653 root NOFD /proc/71653/fd (opendir: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71670 root cwd unknown /proc/71670/cwd (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71670 root rtd unknown /proc/71670/root (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71670 root txt unknown /proc/71670/exe (readlink: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71670 root NOFD /proc/71670/fd (opendir: Permission denied) 2024-03-28T15:55:40.620 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71671 root cwd unknown /proc/71671/cwd (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71671 root rtd unknown /proc/71671/root (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71671 root txt unknown /proc/71671/exe (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71671 root NOFD /proc/71671/fd (opendir: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71688 root cwd unknown /proc/71688/cwd (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71688 root rtd unknown /proc/71688/root (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71688 root txt unknown /proc/71688/exe (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71688 root NOFD /proc/71688/fd (opendir: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71689 root cwd unknown /proc/71689/cwd (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71689 root rtd unknown /proc/71689/root (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71689 root txt unknown /proc/71689/exe (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 71689 root NOFD /proc/71689/fd (opendir: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71706 root cwd unknown /proc/71706/cwd (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71706 root rtd unknown /proc/71706/root (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71706 root txt unknown /proc/71706/exe (readlink: Permission denied) 2024-03-28T15:55:40.621 INFO:tasks.workunit.client.0.smithi023.stdout:journalct 71706 root NOFD /proc/71706/fd (opendir: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72245 root cwd unknown /proc/72245/cwd (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72245 root rtd unknown /proc/72245/root (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72245 root txt unknown /proc/72245/exe (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72245 root NOFD /proc/72245/fd (opendir: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72247 root cwd unknown /proc/72247/cwd (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72247 root rtd unknown /proc/72247/root (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72247 root txt unknown /proc/72247/exe (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72247 root NOFD /proc/72247/fd (opendir: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 ceph cwd unknown /proc/72251/cwd (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 ceph rtd unknown /proc/72251/root (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 ceph txt unknown /proc/72251/exe (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 ceph NOFD /proc/72251/fd (opendir: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72252 log ceph cwd unknown /proc/72251/task/72252/cwd (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72252 log ceph rtd unknown /proc/72251/task/72252/root (readlink: Permission denied) 2024-03-28T15:55:40.622 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72252 log ceph txt unknown /proc/72251/task/72252/exe (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72252 log ceph NOFD /proc/72251/task/72252/fd (opendir: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72254 msgr-work ceph cwd unknown /proc/72251/task/72254/cwd (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72254 msgr-work ceph rtd unknown /proc/72251/task/72254/root (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72254 msgr-work ceph txt unknown /proc/72251/task/72254/exe (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72254 msgr-work ceph NOFD /proc/72251/task/72254/fd (opendir: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72255 msgr-work ceph cwd unknown /proc/72251/task/72255/cwd (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72255 msgr-work ceph rtd unknown /proc/72251/task/72255/root (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72255 msgr-work ceph txt unknown /proc/72251/task/72255/exe (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72255 msgr-work ceph NOFD /proc/72251/task/72255/fd (opendir: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72256 msgr-work ceph cwd unknown /proc/72251/task/72256/cwd (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72256 msgr-work ceph rtd unknown /proc/72251/task/72256/root (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72256 msgr-work ceph txt unknown /proc/72251/task/72256/exe (readlink: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72256 msgr-work ceph NOFD /proc/72251/task/72256/fd (opendir: Permission denied) 2024-03-28T15:55:40.623 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72260 service ceph cwd unknown /proc/72251/task/72260/cwd (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72260 service ceph rtd unknown /proc/72251/task/72260/root (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72260 service ceph txt unknown /proc/72251/task/72260/exe (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72260 service ceph NOFD /proc/72251/task/72260/fd (opendir: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72261 admin_soc ceph cwd unknown /proc/72251/task/72261/cwd (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72261 admin_soc ceph rtd unknown /proc/72251/task/72261/root (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72261 admin_soc ceph txt unknown /proc/72251/task/72261/exe (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72261 admin_soc ceph NOFD /proc/72251/task/72261/fd (opendir: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72262 signal_ha ceph cwd unknown /proc/72251/task/72262/cwd (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72262 signal_ha ceph rtd unknown /proc/72251/task/72262/root (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72262 signal_ha ceph txt unknown /proc/72251/task/72262/exe (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72262 signal_ha ceph NOFD /proc/72251/task/72262/fd (opendir: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72263 io_contex ceph cwd unknown /proc/72251/task/72263/cwd (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72263 io_contex ceph rtd unknown /proc/72251/task/72263/root (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72263 io_contex ceph txt unknown /proc/72251/task/72263/exe (readlink: Permission denied) 2024-03-28T15:55:40.624 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72263 io_contex ceph NOFD /proc/72251/task/72263/fd (opendir: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72264 io_contex ceph cwd unknown /proc/72251/task/72264/cwd (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72264 io_contex ceph rtd unknown /proc/72251/task/72264/root (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72264 io_contex ceph txt unknown /proc/72251/task/72264/exe (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72264 io_contex ceph NOFD /proc/72251/task/72264/fd (opendir: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72265 ms_dispat ceph cwd unknown /proc/72251/task/72265/cwd (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72265 ms_dispat ceph rtd unknown /proc/72251/task/72265/root (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72265 ms_dispat ceph txt unknown /proc/72251/task/72265/exe (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72265 ms_dispat ceph NOFD /proc/72251/task/72265/fd (opendir: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72266 ms_local ceph cwd unknown /proc/72251/task/72266/cwd (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72266 ms_local ceph rtd unknown /proc/72251/task/72266/root (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72266 ms_local ceph txt unknown /proc/72251/task/72266/exe (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72266 ms_local ceph NOFD /proc/72251/task/72266/fd (opendir: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72267 safe_time ceph cwd unknown /proc/72251/task/72267/cwd (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72267 safe_time ceph rtd unknown /proc/72251/task/72267/root (readlink: Permission denied) 2024-03-28T15:55:40.625 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72267 safe_time ceph txt unknown /proc/72251/task/72267/exe (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72267 safe_time ceph NOFD /proc/72251/task/72267/fd (opendir: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72268 safe_time ceph cwd unknown /proc/72251/task/72268/cwd (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72268 safe_time ceph rtd unknown /proc/72251/task/72268/root (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72268 safe_time ceph txt unknown /proc/72251/task/72268/exe (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72268 safe_time ceph NOFD /proc/72251/task/72268/fd (opendir: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72269 ceph-mds ceph cwd unknown /proc/72251/task/72269/cwd (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72269 ceph-mds ceph rtd unknown /proc/72251/task/72269/root (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72269 ceph-mds ceph txt unknown /proc/72251/task/72269/exe (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72269 ceph-mds ceph NOFD /proc/72251/task/72269/fd (opendir: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72270 safe_time ceph cwd unknown /proc/72251/task/72270/cwd (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72270 safe_time ceph rtd unknown /proc/72251/task/72270/root (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72270 safe_time ceph txt unknown /proc/72251/task/72270/exe (readlink: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72251 72270 safe_time ceph NOFD /proc/72251/task/72270/fd (opendir: Permission denied) 2024-03-28T15:55:40.626 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 72588 root cwd unknown /proc/72588/cwd (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 72588 root rtd unknown /proc/72588/root (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 72588 root txt unknown /proc/72588/exe (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 72588 root NOFD /proc/72588/fd (opendir: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72591 root cwd unknown /proc/72591/cwd (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72591 root rtd unknown /proc/72591/root (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72591 root txt unknown /proc/72591/exe (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 72591 root NOFD /proc/72591/fd (opendir: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72593 root cwd unknown /proc/72593/cwd (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72593 root rtd unknown /proc/72593/root (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72593 root txt unknown /proc/72593/exe (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 72593 root NOFD /proc/72593/fd (opendir: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 ceph cwd unknown /proc/72595/cwd (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 ceph rtd unknown /proc/72595/root (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 ceph txt unknown /proc/72595/exe (readlink: Permission denied) 2024-03-28T15:55:40.627 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 ceph NOFD /proc/72595/fd (opendir: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72596 log ceph cwd unknown /proc/72595/task/72596/cwd (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72596 log ceph rtd unknown /proc/72595/task/72596/root (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72596 log ceph txt unknown /proc/72595/task/72596/exe (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72596 log ceph NOFD /proc/72595/task/72596/fd (opendir: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72598 msgr-work ceph cwd unknown /proc/72595/task/72598/cwd (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72598 msgr-work ceph rtd unknown /proc/72595/task/72598/root (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72598 msgr-work ceph txt unknown /proc/72595/task/72598/exe (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72598 msgr-work ceph NOFD /proc/72595/task/72598/fd (opendir: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72599 msgr-work ceph cwd unknown /proc/72595/task/72599/cwd (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72599 msgr-work ceph rtd unknown /proc/72595/task/72599/root (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72599 msgr-work ceph txt unknown /proc/72595/task/72599/exe (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72599 msgr-work ceph NOFD /proc/72595/task/72599/fd (opendir: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72600 msgr-work ceph cwd unknown /proc/72595/task/72600/cwd (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72600 msgr-work ceph rtd unknown /proc/72595/task/72600/root (readlink: Permission denied) 2024-03-28T15:55:40.628 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72600 msgr-work ceph txt unknown /proc/72595/task/72600/exe (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72600 msgr-work ceph NOFD /proc/72595/task/72600/fd (opendir: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72604 service ceph cwd unknown /proc/72595/task/72604/cwd (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72604 service ceph rtd unknown /proc/72595/task/72604/root (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72604 service ceph txt unknown /proc/72595/task/72604/exe (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72604 service ceph NOFD /proc/72595/task/72604/fd (opendir: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72605 admin_soc ceph cwd unknown /proc/72595/task/72605/cwd (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72605 admin_soc ceph rtd unknown /proc/72595/task/72605/root (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72605 admin_soc ceph txt unknown /proc/72595/task/72605/exe (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72605 admin_soc ceph NOFD /proc/72595/task/72605/fd (opendir: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72606 signal_ha ceph cwd unknown /proc/72595/task/72606/cwd (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72606 signal_ha ceph rtd unknown /proc/72595/task/72606/root (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72606 signal_ha ceph txt unknown /proc/72595/task/72606/exe (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72606 signal_ha ceph NOFD /proc/72595/task/72606/fd (opendir: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72607 io_contex ceph cwd unknown /proc/72595/task/72607/cwd (readlink: Permission denied) 2024-03-28T15:55:40.629 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72607 io_contex ceph rtd unknown /proc/72595/task/72607/root (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72607 io_contex ceph txt unknown /proc/72595/task/72607/exe (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72607 io_contex ceph NOFD /proc/72595/task/72607/fd (opendir: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72608 io_contex ceph cwd unknown /proc/72595/task/72608/cwd (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72608 io_contex ceph rtd unknown /proc/72595/task/72608/root (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72608 io_contex ceph txt unknown /proc/72595/task/72608/exe (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72608 io_contex ceph NOFD /proc/72595/task/72608/fd (opendir: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72609 ms_dispat ceph cwd unknown /proc/72595/task/72609/cwd (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72609 ms_dispat ceph rtd unknown /proc/72595/task/72609/root (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72609 ms_dispat ceph txt unknown /proc/72595/task/72609/exe (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72609 ms_dispat ceph NOFD /proc/72595/task/72609/fd (opendir: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72610 ms_local ceph cwd unknown /proc/72595/task/72610/cwd (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72610 ms_local ceph rtd unknown /proc/72595/task/72610/root (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72610 ms_local ceph txt unknown /proc/72595/task/72610/exe (readlink: Permission denied) 2024-03-28T15:55:40.630 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72610 ms_local ceph NOFD /proc/72595/task/72610/fd (opendir: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72611 safe_time ceph cwd unknown /proc/72595/task/72611/cwd (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72611 safe_time ceph rtd unknown /proc/72595/task/72611/root (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72611 safe_time ceph txt unknown /proc/72595/task/72611/exe (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72611 safe_time ceph NOFD /proc/72595/task/72611/fd (opendir: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72612 safe_time ceph cwd unknown /proc/72595/task/72612/cwd (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72612 safe_time ceph rtd unknown /proc/72595/task/72612/root (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72612 safe_time ceph txt unknown /proc/72595/task/72612/exe (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72612 safe_time ceph NOFD /proc/72595/task/72612/fd (opendir: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72613 ceph-mds ceph cwd unknown /proc/72595/task/72613/cwd (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72613 ceph-mds ceph rtd unknown /proc/72595/task/72613/root (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72613 ceph-mds ceph txt unknown /proc/72595/task/72613/exe (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72613 ceph-mds ceph NOFD /proc/72595/task/72613/fd (opendir: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72618 safe_time ceph cwd unknown /proc/72595/task/72618/cwd (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72618 safe_time ceph rtd unknown /proc/72595/task/72618/root (readlink: Permission denied) 2024-03-28T15:55:40.631 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72618 safe_time ceph txt unknown /proc/72595/task/72618/exe (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 72595 72618 safe_time ceph NOFD /proc/72595/task/72618/fd (opendir: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73186 root cwd unknown /proc/73186/cwd (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73186 root rtd unknown /proc/73186/root (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73186 root txt unknown /proc/73186/exe (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73186 root NOFD /proc/73186/fd (opendir: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73188 root cwd unknown /proc/73188/cwd (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73188 root rtd unknown /proc/73188/root (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73188 root txt unknown /proc/73188/exe (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73188 root NOFD /proc/73188/fd (opendir: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 ceph cwd unknown /proc/73190/cwd (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 ceph rtd unknown /proc/73190/root (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 ceph txt unknown /proc/73190/exe (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 ceph NOFD /proc/73190/fd (opendir: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73191 log ceph cwd unknown /proc/73190/task/73191/cwd (readlink: Permission denied) 2024-03-28T15:55:40.632 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73191 log ceph rtd unknown /proc/73190/task/73191/root (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73191 log ceph txt unknown /proc/73190/task/73191/exe (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73191 log ceph NOFD /proc/73190/task/73191/fd (opendir: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73193 msgr-work ceph cwd unknown /proc/73190/task/73193/cwd (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73193 msgr-work ceph rtd unknown /proc/73190/task/73193/root (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73193 msgr-work ceph txt unknown /proc/73190/task/73193/exe (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73193 msgr-work ceph NOFD /proc/73190/task/73193/fd (opendir: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73194 msgr-work ceph cwd unknown /proc/73190/task/73194/cwd (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73194 msgr-work ceph rtd unknown /proc/73190/task/73194/root (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73194 msgr-work ceph txt unknown /proc/73190/task/73194/exe (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73194 msgr-work ceph NOFD /proc/73190/task/73194/fd (opendir: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73195 msgr-work ceph cwd unknown /proc/73190/task/73195/cwd (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73195 msgr-work ceph rtd unknown /proc/73190/task/73195/root (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73195 msgr-work ceph txt unknown /proc/73190/task/73195/exe (readlink: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73195 msgr-work ceph NOFD /proc/73190/task/73195/fd (opendir: Permission denied) 2024-03-28T15:55:40.633 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73199 service ceph cwd unknown /proc/73190/task/73199/cwd (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73199 service ceph rtd unknown /proc/73190/task/73199/root (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73199 service ceph txt unknown /proc/73190/task/73199/exe (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73199 service ceph NOFD /proc/73190/task/73199/fd (opendir: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73200 admin_soc ceph cwd unknown /proc/73190/task/73200/cwd (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73200 admin_soc ceph rtd unknown /proc/73190/task/73200/root (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73200 admin_soc ceph txt unknown /proc/73190/task/73200/exe (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73200 admin_soc ceph NOFD /proc/73190/task/73200/fd (opendir: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73201 signal_ha ceph cwd unknown /proc/73190/task/73201/cwd (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73201 signal_ha ceph rtd unknown /proc/73190/task/73201/root (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73201 signal_ha ceph txt unknown /proc/73190/task/73201/exe (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73201 signal_ha ceph NOFD /proc/73190/task/73201/fd (opendir: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73202 io_contex ceph cwd unknown /proc/73190/task/73202/cwd (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73202 io_contex ceph rtd unknown /proc/73190/task/73202/root (readlink: Permission denied) 2024-03-28T15:55:40.634 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73202 io_contex ceph txt unknown /proc/73190/task/73202/exe (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73202 io_contex ceph NOFD /proc/73190/task/73202/fd (opendir: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73203 io_contex ceph cwd unknown /proc/73190/task/73203/cwd (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73203 io_contex ceph rtd unknown /proc/73190/task/73203/root (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73203 io_contex ceph txt unknown /proc/73190/task/73203/exe (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73203 io_contex ceph NOFD /proc/73190/task/73203/fd (opendir: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73204 ms_dispat ceph cwd unknown /proc/73190/task/73204/cwd (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73204 ms_dispat ceph rtd unknown /proc/73190/task/73204/root (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73204 ms_dispat ceph txt unknown /proc/73190/task/73204/exe (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73204 ms_dispat ceph NOFD /proc/73190/task/73204/fd (opendir: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73205 ms_local ceph cwd unknown /proc/73190/task/73205/cwd (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73205 ms_local ceph rtd unknown /proc/73190/task/73205/root (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73205 ms_local ceph txt unknown /proc/73190/task/73205/exe (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73205 ms_local ceph NOFD /proc/73190/task/73205/fd (opendir: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73206 safe_time ceph cwd unknown /proc/73190/task/73206/cwd (readlink: Permission denied) 2024-03-28T15:55:40.635 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73206 safe_time ceph rtd unknown /proc/73190/task/73206/root (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73206 safe_time ceph txt unknown /proc/73190/task/73206/exe (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73206 safe_time ceph NOFD /proc/73190/task/73206/fd (opendir: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73207 safe_time ceph cwd unknown /proc/73190/task/73207/cwd (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73207 safe_time ceph rtd unknown /proc/73190/task/73207/root (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73207 safe_time ceph txt unknown /proc/73190/task/73207/exe (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73207 safe_time ceph NOFD /proc/73190/task/73207/fd (opendir: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73208 ceph-mds ceph cwd unknown /proc/73190/task/73208/cwd (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73208 ceph-mds ceph rtd unknown /proc/73190/task/73208/root (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73208 ceph-mds ceph txt unknown /proc/73190/task/73208/exe (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73208 ceph-mds ceph NOFD /proc/73190/task/73208/fd (opendir: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73338 safe_time ceph cwd unknown /proc/73190/task/73338/cwd (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73338 safe_time ceph rtd unknown /proc/73190/task/73338/root (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73338 safe_time ceph txt unknown /proc/73190/task/73338/exe (readlink: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 73338 safe_time ceph NOFD /proc/73190/task/73338/fd (opendir: Permission denied) 2024-03-28T15:55:40.636 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76795 ceph_time ceph cwd unknown /proc/73190/task/76795/cwd (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76795 ceph_time ceph rtd unknown /proc/73190/task/76795/root (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76795 ceph_time ceph txt unknown /proc/73190/task/76795/exe (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76795 ceph_time ceph NOFD /proc/73190/task/76795/fd (opendir: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76796 OpHistory ceph cwd unknown /proc/73190/task/76796/cwd (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76796 OpHistory ceph rtd unknown /proc/73190/task/76796/root (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76796 OpHistory ceph txt unknown /proc/73190/task/76796/exe (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76796 OpHistory ceph NOFD /proc/73190/task/76796/fd (opendir: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76797 ms_dispat ceph cwd unknown /proc/73190/task/76797/cwd (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76797 ms_dispat ceph rtd unknown /proc/73190/task/76797/root (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76797 ms_dispat ceph txt unknown /proc/73190/task/76797/exe (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76797 ms_dispat ceph NOFD /proc/73190/task/76797/fd (opendir: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76798 ms_dispat ceph cwd unknown /proc/73190/task/76798/cwd (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76798 ms_dispat ceph rtd unknown /proc/73190/task/76798/root (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76798 ms_dispat ceph txt unknown /proc/73190/task/76798/exe (readlink: Permission denied) 2024-03-28T15:55:40.637 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76798 ms_dispat ceph NOFD /proc/73190/task/76798/fd (opendir: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76799 mds_rank_ ceph cwd unknown /proc/73190/task/76799/cwd (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76799 mds_rank_ ceph rtd unknown /proc/73190/task/76799/root (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76799 mds_rank_ ceph txt unknown /proc/73190/task/76799/exe (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76799 mds_rank_ ceph NOFD /proc/73190/task/76799/fd (opendir: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76800 PQ_Finish ceph cwd unknown /proc/73190/task/76800/cwd (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76800 PQ_Finish ceph rtd unknown /proc/73190/task/76800/root (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76800 PQ_Finish ceph txt unknown /proc/73190/task/76800/exe (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76800 PQ_Finish ceph NOFD /proc/73190/task/76800/fd (opendir: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76801 safe_time ceph cwd unknown /proc/73190/task/76801/cwd (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76801 safe_time ceph rtd unknown /proc/73190/task/76801/root (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76801 safe_time ceph txt unknown /proc/73190/task/76801/exe (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76801 safe_time ceph NOFD /proc/73190/task/76801/fd (opendir: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76802 MR_Finish ceph cwd unknown /proc/73190/task/76802/cwd (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76802 MR_Finish ceph rtd unknown /proc/73190/task/76802/root (readlink: Permission denied) 2024-03-28T15:55:40.638 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76802 MR_Finish ceph txt unknown /proc/73190/task/76802/exe (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76802 MR_Finish ceph NOFD /proc/73190/task/76802/fd (opendir: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76804 md_submit ceph cwd unknown /proc/73190/task/76804/cwd (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76804 md_submit ceph rtd unknown /proc/73190/task/76804/root (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76804 md_submit ceph txt unknown /proc/73190/task/76804/exe (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73190 76804 md_submit ceph NOFD /proc/73190/task/76804/fd (opendir: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73548 root cwd unknown /proc/73548/cwd (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73548 root rtd unknown /proc/73548/root (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73548 root txt unknown /proc/73548/exe (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:conmon 73548 root NOFD /proc/73548/fd (opendir: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73550 root cwd unknown /proc/73550/cwd (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73550 root rtd unknown /proc/73550/root (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73550 root txt unknown /proc/73550/exe (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:podman-in 73550 root NOFD /proc/73550/fd (opendir: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 ceph cwd unknown /proc/73552/cwd (readlink: Permission denied) 2024-03-28T15:55:40.639 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 ceph rtd unknown /proc/73552/root (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 ceph txt unknown /proc/73552/exe (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 ceph NOFD /proc/73552/fd (opendir: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73553 log ceph cwd unknown /proc/73552/task/73553/cwd (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73553 log ceph rtd unknown /proc/73552/task/73553/root (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73553 log ceph txt unknown /proc/73552/task/73553/exe (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73553 log ceph NOFD /proc/73552/task/73553/fd (opendir: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73555 msgr-work ceph cwd unknown /proc/73552/task/73555/cwd (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73555 msgr-work ceph rtd unknown /proc/73552/task/73555/root (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73555 msgr-work ceph txt unknown /proc/73552/task/73555/exe (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73555 msgr-work ceph NOFD /proc/73552/task/73555/fd (opendir: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73556 msgr-work ceph cwd unknown /proc/73552/task/73556/cwd (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73556 msgr-work ceph rtd unknown /proc/73552/task/73556/root (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73556 msgr-work ceph txt unknown /proc/73552/task/73556/exe (readlink: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73556 msgr-work ceph NOFD /proc/73552/task/73556/fd (opendir: Permission denied) 2024-03-28T15:55:40.640 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73557 msgr-work ceph cwd unknown /proc/73552/task/73557/cwd (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73557 msgr-work ceph rtd unknown /proc/73552/task/73557/root (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73557 msgr-work ceph txt unknown /proc/73552/task/73557/exe (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73557 msgr-work ceph NOFD /proc/73552/task/73557/fd (opendir: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73561 service ceph cwd unknown /proc/73552/task/73561/cwd (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73561 service ceph rtd unknown /proc/73552/task/73561/root (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73561 service ceph txt unknown /proc/73552/task/73561/exe (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73561 service ceph NOFD /proc/73552/task/73561/fd (opendir: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73562 admin_soc ceph cwd unknown /proc/73552/task/73562/cwd (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73562 admin_soc ceph rtd unknown /proc/73552/task/73562/root (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73562 admin_soc ceph txt unknown /proc/73552/task/73562/exe (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73562 admin_soc ceph NOFD /proc/73552/task/73562/fd (opendir: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73563 signal_ha ceph cwd unknown /proc/73552/task/73563/cwd (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73563 signal_ha ceph rtd unknown /proc/73552/task/73563/root (readlink: Permission denied) 2024-03-28T15:55:40.641 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73563 signal_ha ceph txt unknown /proc/73552/task/73563/exe (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73563 signal_ha ceph NOFD /proc/73552/task/73563/fd (opendir: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73564 io_contex ceph cwd unknown /proc/73552/task/73564/cwd (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73564 io_contex ceph rtd unknown /proc/73552/task/73564/root (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73564 io_contex ceph txt unknown /proc/73552/task/73564/exe (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73564 io_contex ceph NOFD /proc/73552/task/73564/fd (opendir: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73565 io_contex ceph cwd unknown /proc/73552/task/73565/cwd (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73565 io_contex ceph rtd unknown /proc/73552/task/73565/root (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73565 io_contex ceph txt unknown /proc/73552/task/73565/exe (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73565 io_contex ceph NOFD /proc/73552/task/73565/fd (opendir: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73566 ms_dispat ceph cwd unknown /proc/73552/task/73566/cwd (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73566 ms_dispat ceph rtd unknown /proc/73552/task/73566/root (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73566 ms_dispat ceph txt unknown /proc/73552/task/73566/exe (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73566 ms_dispat ceph NOFD /proc/73552/task/73566/fd (opendir: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73567 ms_local ceph cwd unknown /proc/73552/task/73567/cwd (readlink: Permission denied) 2024-03-28T15:55:40.642 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73567 ms_local ceph rtd unknown /proc/73552/task/73567/root (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73567 ms_local ceph txt unknown /proc/73552/task/73567/exe (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73567 ms_local ceph NOFD /proc/73552/task/73567/fd (opendir: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73568 safe_time ceph cwd unknown /proc/73552/task/73568/cwd (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73568 safe_time ceph rtd unknown /proc/73552/task/73568/root (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73568 safe_time ceph txt unknown /proc/73552/task/73568/exe (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73568 safe_time ceph NOFD /proc/73552/task/73568/fd (opendir: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73569 safe_time ceph cwd unknown /proc/73552/task/73569/cwd (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73569 safe_time ceph rtd unknown /proc/73552/task/73569/root (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73569 safe_time ceph txt unknown /proc/73552/task/73569/exe (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73569 safe_time ceph NOFD /proc/73552/task/73569/fd (opendir: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73570 ceph-mds ceph cwd unknown /proc/73552/task/73570/cwd (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73570 ceph-mds ceph rtd unknown /proc/73552/task/73570/root (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73570 ceph-mds ceph txt unknown /proc/73552/task/73570/exe (readlink: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73570 ceph-mds ceph NOFD /proc/73552/task/73570/fd (opendir: Permission denied) 2024-03-28T15:55:40.643 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73575 safe_time ceph cwd unknown /proc/73552/task/73575/cwd (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73575 safe_time ceph rtd unknown /proc/73552/task/73575/root (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73575 safe_time ceph txt unknown /proc/73552/task/73575/exe (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 73575 safe_time ceph NOFD /proc/73552/task/73575/fd (opendir: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76661 ceph_time ceph cwd unknown /proc/73552/task/76661/cwd (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76661 ceph_time ceph rtd unknown /proc/73552/task/76661/root (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76661 ceph_time ceph txt unknown /proc/73552/task/76661/exe (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76661 ceph_time ceph NOFD /proc/73552/task/76661/fd (opendir: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76662 OpHistory ceph cwd unknown /proc/73552/task/76662/cwd (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76662 OpHistory ceph rtd unknown /proc/73552/task/76662/root (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76662 OpHistory ceph txt unknown /proc/73552/task/76662/exe (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76662 OpHistory ceph NOFD /proc/73552/task/76662/fd (opendir: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76663 ms_dispat ceph cwd unknown /proc/73552/task/76663/cwd (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76663 ms_dispat ceph rtd unknown /proc/73552/task/76663/root (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76663 ms_dispat ceph txt unknown /proc/73552/task/76663/exe (readlink: Permission denied) 2024-03-28T15:55:40.644 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76663 ms_dispat ceph NOFD /proc/73552/task/76663/fd (opendir: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76664 ms_dispat ceph cwd unknown /proc/73552/task/76664/cwd (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76664 ms_dispat ceph rtd unknown /proc/73552/task/76664/root (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76664 ms_dispat ceph txt unknown /proc/73552/task/76664/exe (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76664 ms_dispat ceph NOFD /proc/73552/task/76664/fd (opendir: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76665 mds_rank_ ceph cwd unknown /proc/73552/task/76665/cwd (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76665 mds_rank_ ceph rtd unknown /proc/73552/task/76665/root (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76665 mds_rank_ ceph txt unknown /proc/73552/task/76665/exe (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76665 mds_rank_ ceph NOFD /proc/73552/task/76665/fd (opendir: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76666 PQ_Finish ceph cwd unknown /proc/73552/task/76666/cwd (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76666 PQ_Finish ceph rtd unknown /proc/73552/task/76666/root (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76666 PQ_Finish ceph txt unknown /proc/73552/task/76666/exe (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76666 PQ_Finish ceph NOFD /proc/73552/task/76666/fd (opendir: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76667 safe_time ceph cwd unknown /proc/73552/task/76667/cwd (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76667 safe_time ceph rtd unknown /proc/73552/task/76667/root (readlink: Permission denied) 2024-03-28T15:55:40.645 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76667 safe_time ceph txt unknown /proc/73552/task/76667/exe (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76667 safe_time ceph NOFD /proc/73552/task/76667/fd (opendir: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76668 MR_Finish ceph cwd unknown /proc/73552/task/76668/cwd (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76668 MR_Finish ceph rtd unknown /proc/73552/task/76668/root (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76668 MR_Finish ceph txt unknown /proc/73552/task/76668/exe (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76668 MR_Finish ceph NOFD /proc/73552/task/76668/fd (opendir: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76669 md_submit ceph cwd unknown /proc/73552/task/76669/cwd (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76669 md_submit ceph rtd unknown /proc/73552/task/76669/root (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76669 md_submit ceph txt unknown /proc/73552/task/76669/exe (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76669 md_submit ceph NOFD /proc/73552/task/76669/fd (opendir: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76792 ms_dispat ceph cwd unknown /proc/73552/task/76792/cwd (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76792 ms_dispat ceph rtd unknown /proc/73552/task/76792/root (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76792 ms_dispat ceph txt unknown /proc/73552/task/76792/exe (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76792 ms_dispat ceph NOFD /proc/73552/task/76792/fd (opendir: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76793 quiesce.a ceph cwd unknown /proc/73552/task/76793/cwd (readlink: Permission denied) 2024-03-28T15:55:40.646 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76793 quiesce.a ceph rtd unknown /proc/73552/task/76793/root (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76793 quiesce.a ceph txt unknown /proc/73552/task/76793/exe (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76793 quiesce.a ceph NOFD /proc/73552/task/76793/fd (opendir: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76794 quiesce_d ceph cwd unknown /proc/73552/task/76794/cwd (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76794 quiesce_d ceph rtd unknown /proc/73552/task/76794/root (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76794 quiesce_d ceph txt unknown /proc/73552/task/76794/exe (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-mds 73552 76794 quiesce_d ceph NOFD /proc/73552/task/76794/fd (opendir: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 76158 root cwd unknown /proc/76158/cwd (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 76158 root rtd unknown /proc/76158/root (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 76158 root txt unknown /proc/76158/exe (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 76158 root NOFD /proc/76158/fd (opendir: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 78461 root cwd unknown /proc/78461/cwd (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 78461 root rtd unknown /proc/78461/root (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 78461 root txt unknown /proc/78461/exe (readlink: Permission denied) 2024-03-28T15:55:40.647 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/7 78461 root NOFD /proc/78461/fd (opendir: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 78969 root cwd unknown /proc/78969/cwd (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 78969 root rtd unknown /proc/78969/root (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 78969 root txt unknown /proc/78969/exe (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/5 78969 root NOFD /proc/78969/fd (opendir: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 79666 root cwd unknown /proc/79666/cwd (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 79666 root rtd unknown /proc/79666/root (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 79666 root txt unknown /proc/79666/exe (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/u 79666 root NOFD /proc/79666/fd (opendir: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 80553 root cwd unknown /proc/80553/cwd (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 80553 root rtd unknown /proc/80553/root (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 80553 root txt unknown /proc/80553/exe (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/4 80553 root NOFD /proc/80553/fd (opendir: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 82213 root cwd unknown /proc/82213/cwd (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 82213 root rtd unknown /proc/82213/root (readlink: Permission denied) 2024-03-28T15:55:40.648 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 82213 root txt unknown /proc/82213/exe (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/6 82213 root NOFD /proc/82213/fd (opendir: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 85186 root cwd unknown /proc/85186/cwd (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 85186 root rtd unknown /proc/85186/root (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 85186 root txt unknown /proc/85186/exe (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/0 85186 root NOFD /proc/85186/fd (opendir: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-msgr 85669 root cwd unknown /proc/85669/cwd (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-msgr 85669 root rtd unknown /proc/85669/root (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-msgr 85669 root txt unknown /proc/85669/exe (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-msgr 85669 root NOFD /proc/85669/fd (opendir: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-watc 85670 root cwd unknown /proc/85670/cwd (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-watc 85670 root rtd unknown /proc/85670/root (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-watc 85670 root txt unknown /proc/85670/exe (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-watc 85670 root NOFD /proc/85670/fd (opendir: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-comp 85671 root cwd unknown /proc/85671/cwd (readlink: Permission denied) 2024-03-28T15:55:40.649 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-comp 85671 root rtd unknown /proc/85671/root (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-comp 85671 root txt unknown /proc/85671/exe (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:ceph-comp 85671 root NOFD /proc/85671/fd (opendir: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86167 root cwd unknown /proc/86167/cwd (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86167 root rtd unknown /proc/86167/root (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86167 root txt unknown /proc/86167/exe (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86167 root NOFD /proc/86167/fd (opendir: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86227 root cwd unknown /proc/86227/cwd (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86227 root rtd unknown /proc/86227/root (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86227 root txt unknown /proc/86227/exe (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sshd 86227 root NOFD /proc/86227/fd (opendir: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 86809 root cwd unknown /proc/86809/cwd (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 86809 root rtd unknown /proc/86809/root (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 86809 root txt unknown /proc/86809/exe (readlink: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:sudo 86809 root NOFD /proc/86809/fd (opendir: Permission denied) 2024-03-28T15:55:40.650 INFO:tasks.workunit.client.0.smithi023.stdout:daemon-he 86839 root cwd unknown /proc/86839/cwd (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:daemon-he 86839 root rtd unknown /proc/86839/root (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:daemon-he 86839 root txt unknown /proc/86839/exe (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:daemon-he 86839 root NOFD /proc/86839/fd (opendir: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:bash 86843 root cwd unknown /proc/86843/cwd (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:bash 86843 root rtd unknown /proc/86843/root (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:bash 86843 root txt unknown /proc/86843/exe (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:bash 86843 root NOFD /proc/86843/fd (opendir: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 88351 root cwd unknown /proc/88351/cwd (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 88351 root rtd unknown /proc/88351/root (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 88351 root txt unknown /proc/88351/exe (readlink: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:kworker/3 88351 root NOFD /proc/88351/fd (opendir: Permission denied) 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu cwd DIR 0,119 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu rtd DIR 8,1 4096 2 / 2024-03-28T15:55:40.651 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu txt REG 8,1 40944 10098 /usr/bin/timeout 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-28T15:55:40.652 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu 0r FIFO 0,13 0t0 238809 pipe 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu 1w FIFO 0,13 0t0 238810 pipe 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:timeout 88538 ubuntu 2w FIFO 0,13 0t0 238811 pipe 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu cwd DIR 0,119 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu rtd DIR 8,1 4096 2 / 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu txt REG 8,1 1388920 9667 /usr/bin/bash 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 2586930 17848 /usr/lib/locale/en_US.utf8/LC_COLLATE 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 54 17851 /usr/lib/locale/en_US.utf8/LC_NUMERIC 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 3284 30787 /usr/lib/locale/en_US.utf8/LC_TIME 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 286 30784 /usr/lib/locale/en_US.utf8/LC_MONETARY 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 57 28351 /usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 34 30837 /usr/lib/locale/en_US.utf8/LC_PAPER 2024-03-28T15:55:40.653 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 77 17850 /usr/lib/locale/en_US.utf8/LC_NAME 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 167 30779 /usr/lib/locale/en_US.utf8/LC_ADDRESS 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 59 30786 /usr/lib/locale/en_US.utf8/LC_TELEPHONE 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 195080 9664 /usr/lib64/libtinfo.so.6.2 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 23 30783 /usr/lib/locale/en_US.utf8/LC_MEASUREMENT 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu mem REG 8,1 369 30782 /usr/lib/locale/en_US.utf8/LC_IDENTIFICATION 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu 0r FIFO 0,13 0t0 238809 pipe 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu 1w FIFO 0,13 0t0 238810 pipe 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu 2w FIFO 0,13 0t0 238811 pipe 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:fsync-tes 88586 ubuntu 255r REG 8,1 207 531148 /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsync-tester.sh 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:sleep 88926 root cwd unknown /proc/88926/cwd (readlink: Permission denied) 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:sleep 88926 root rtd unknown /proc/88926/root (readlink: Permission denied) 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:sleep 88926 root txt unknown /proc/88926/exe (readlink: Permission denied) 2024-03-28T15:55:40.654 INFO:tasks.workunit.client.0.smithi023.stdout:sleep 88926 root NOFD /proc/88926/fd (opendir: Permission denied) 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu cwd DIR 0,119 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu rtd DIR 8,1 4096 2 / 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-28T15:55:40.655 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 0r FIFO 0,13 0t0 238809 pipe 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 1w FIFO 0,13 0t0 238810 pipe 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 2w FIFO 0,13 0t0 238811 pipe 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 3r DIR 0,20 0 1 /proc 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 4r DIR 0,20 0 236946 /proc/88928/fd 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 5w FIFO 0,13 0t0 236951 pipe 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88928 ubuntu 6r FIFO 0,13 0t0 236952 pipe 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu cwd DIR 0,119 4 1099511627782 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu rtd DIR 8,1 4096 2 / 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu txt REG 8,1 179232 12343 /usr/bin/lsof 2024-03-28T15:55:40.656 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 4487176 2250 /usr/lib64/libcrypto.so.3.0.7 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 346132 30861 /usr/lib/locale/C.utf8/LC_CTYPE 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 102552 9686 /usr/lib64/libz.so.1.2.11 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 906256 11403 /usr/lib64/libkrb5.so.3.3 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 2387016 9624 /usr/lib64/libc.so.6 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 70512 9633 /usr/lib64/libresolv.so.2 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 23992 9760 /usr/lib64/libkeyutils.so.1.10 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 66192 11405 /usr/lib64/libkrb5support.so.0.1 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 24400 9711 /usr/lib64/libcom_err.so.2.1 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 98992 11397 /usr/lib64/libk5crypto.so.3.1 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 358448 11393 /usr/lib64/libgssapi_krb5.so.2.2 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 637872 9740 /usr/lib64/libpcre2-8.so.0.11.0 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 196112 11431 /usr/lib64/libtirpc.so.3.0.0 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 175552 9824 /usr/lib64/libselinux.so.1 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 26988 29960 /usr/lib64/gconv/gconv-modules.cache 2024-03-28T15:55:40.657 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu mem REG 8,1 844808 9620 /usr/lib64/ld-linux-x86-64.so.2 2024-03-28T15:55:40.658 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu 4r FIFO 0,13 0t0 236951 pipe 2024-03-28T15:55:40.658 INFO:tasks.workunit.client.0.smithi023.stdout:lsof 88929 ubuntu 7w FIFO 0,13 0t0 236952 pipe 2024-03-28T15:55:40.658 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-28T15:55:40.658 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-28T15:55:40.710 INFO:tasks.workunit:Stopping ['suites/fsync-tester.sh'] on client.0... 2024-03-28T15:55:40.710 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-28T15:55:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:40 smithi117 ceph-mon[36002]: pgmap v80: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 869 KiB/s wr, 13 op/s 2024-03-28T15:55:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:40 smithi120 ceph-mon[36826]: pgmap v80: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 869 KiB/s wr, 13 op/s 2024-03-28T15:55:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:40 smithi023 ceph-mon[32448]: pgmap v80: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 869 KiB/s wr, 13 op/s 2024-03-28T15:55:40.984 DEBUG:teuthology.parallel:result is None 2024-03-28T15:55:40.985 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T15:55:41.019 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-28T15:55:41.019 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-28T15:55:41.031 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:55:41.066 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-28T15:55:41.078 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-28T15:55:41.078 INFO:teuthology.task.exec:Executing custom commands... 2024-03-28T15:55:41.078 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T15:55:41.078 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-28T15:55:41.135 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-28T15:55:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:42 smithi117 ceph-mon[36002]: pgmap v81: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-03-28T15:55:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:42 smithi120 ceph-mon[36826]: pgmap v81: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-03-28T15:55:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:42 smithi023 ceph-mon[32448]: pgmap v81: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.2 MiB/s wr, 19 op/s 2024-03-28T15:55:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:44 smithi117 ceph-mon[36002]: pgmap v82: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:44 smithi120 ceph-mon[36826]: pgmap v82: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:44 smithi023 ceph-mon[32448]: pgmap v82: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:46 smithi117 ceph-mon[36002]: pgmap v83: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:46 smithi120 ceph-mon[36826]: pgmap v83: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:46 smithi023 ceph-mon[32448]: pgmap v83: 129 pgs: 129 active+clean; 310 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 785 KiB/s wr, 13 op/s 2024-03-28T15:55:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:48 smithi117 ceph-mon[36002]: pgmap v84: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 16 MiB/s wr, 22 op/s 2024-03-28T15:55:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:48 smithi120 ceph-mon[36826]: pgmap v84: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 16 MiB/s wr, 22 op/s 2024-03-28T15:55:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:48 smithi023 ceph-mon[32448]: pgmap v84: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 16 MiB/s wr, 22 op/s 2024-03-28T15:55:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:50 smithi117 ceph-mon[36002]: pgmap v85: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 14 op/s 2024-03-28T15:55:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:50 smithi120 ceph-mon[36826]: pgmap v85: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 14 op/s 2024-03-28T15:55:50.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:50 smithi023 ceph-mon[32448]: pgmap v85: 129 pgs: 129 active+clean; 455 MiB data, 598 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 15 MiB/s wr, 14 op/s 2024-03-28T15:55:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:52 smithi117 ceph-mon[36002]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 27 op/s 2024-03-28T15:55:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:52 smithi120 ceph-mon[36826]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 27 op/s 2024-03-28T15:55:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:52 smithi023 ceph-mon[32448]: pgmap v86: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 MiB/s wr, 27 op/s 2024-03-28T15:55:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:54 smithi117 ceph-mon[36002]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:54 smithi120 ceph-mon[36826]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:54 smithi023 ceph-mon[32448]: pgmap v87: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:56 smithi117 ceph-mon[36002]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:56 smithi120 ceph-mon[36826]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:56 smithi023 ceph-mon[32448]: pgmap v88: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 21 MiB/s wr, 21 op/s 2024-03-28T15:55:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:55:58 smithi117 ceph-mon[36002]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 26 op/s 2024-03-28T15:55:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:55:58 smithi120 ceph-mon[36826]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 26 op/s 2024-03-28T15:55:58.924 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:55:58 smithi023 ceph-mon[32448]: pgmap v89: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 21 MiB/s wr, 26 op/s 2024-03-28T15:56:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:00 smithi117 ceph-mon[36002]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.3 MiB/s wr, 17 op/s 2024-03-28T15:56:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:00 smithi120 ceph-mon[36826]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.3 MiB/s wr, 17 op/s 2024-03-28T15:56:00.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:00 smithi023 ceph-mon[32448]: pgmap v90: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.3 MiB/s wr, 17 op/s 2024-03-28T15:56:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:02 smithi117 ceph-mon[36002]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.4 MiB/s wr, 37 op/s 2024-03-28T15:56:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:02 smithi120 ceph-mon[36826]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.4 MiB/s wr, 37 op/s 2024-03-28T15:56:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:02 smithi023 ceph-mon[32448]: pgmap v91: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 6.4 MiB/s wr, 37 op/s 2024-03-28T15:56:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:04 smithi117 ceph-mon[36002]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:04 smithi120 ceph-mon[36826]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:04.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:04 smithi023 ceph-mon[32448]: pgmap v92: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:06 smithi117 ceph-mon[36002]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:06 smithi120 ceph-mon[36826]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:06 smithi023 ceph-mon[32448]: pgmap v93: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:08 smithi117 ceph-mon[36002]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:08 smithi120 ceph-mon[36826]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:08.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:08 smithi023 ceph-mon[32448]: pgmap v94: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:09 smithi120 ceph-mon[36826]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:09 smithi023 ceph-mon[32448]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:10.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:09 smithi117 ceph-mon[36002]: pgmap v95: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-03-28T15:56:12.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:12 smithi117 ceph-mon[36002]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:12.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:12 smithi120 ceph-mon[36826]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:12 smithi023 ceph-mon[32448]: pgmap v96: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 24 KiB/s wr, 28 op/s 2024-03-28T15:56:13.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:13 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:13.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:13 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:13.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:13 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:56:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:14 smithi117 ceph-mon[36002]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:14.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:14 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:56:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:14 smithi120 ceph-mon[36826]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:14.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:14 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:56:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:14 smithi023 ceph-mon[32448]: pgmap v97: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:56:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:56:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:14 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:56:16.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:16 smithi117 ceph-mon[36002]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:16.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:16 smithi120 ceph-mon[36826]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:16 smithi023 ceph-mon[32448]: pgmap v98: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2024-03-28T15:56:18.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:18 smithi117 ceph-mon[36002]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-03-28T15:56:18.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:18 smithi120 ceph-mon[36826]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-03-28T15:56:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:18 smithi023 ceph-mon[32448]: pgmap v99: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.1 KiB/s wr, 13 op/s 2024-03-28T15:56:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:20 smithi117 ceph-mon[36002]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:56:20.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:20 smithi120 ceph-mon[36826]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:56:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:20 smithi023 ceph-mon[32448]: pgmap v100: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 8 op/s 2024-03-28T15:56:22.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:22 smithi117 ceph-mon[36002]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-03-28T15:56:22.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:22 smithi120 ceph-mon[36826]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-03-28T15:56:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:22 smithi023 ceph-mon[32448]: pgmap v101: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.4 KiB/s wr, 13 op/s 2024-03-28T15:56:24.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:24 smithi117 ceph-mon[36002]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:24.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:24 smithi120 ceph-mon[36826]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:24 smithi023 ceph-mon[32448]: pgmap v102: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:26 smithi117 ceph-mon[36002]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:26 smithi120 ceph-mon[36826]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:26 smithi023 ceph-mon[32448]: pgmap v103: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:28.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:28 smithi117 ceph-mon[36002]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:28 smithi120 ceph-mon[36826]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:28 smithi023 ceph-mon[32448]: pgmap v104: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:30.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:30 smithi117 ceph-mon[36002]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:30.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:30 smithi120 ceph-mon[36826]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:30 smithi023 ceph-mon[32448]: pgmap v105: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 511 B/s wr, 8 op/s 2024-03-28T15:56:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:32 smithi117 ceph-mon[36002]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:32 smithi120 ceph-mon[36826]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:32.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:32 smithi023 ceph-mon[32448]: pgmap v106: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-28T15:56:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:34 smithi117 ceph-mon[36002]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:34 smithi120 ceph-mon[36826]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:34 smithi023 ceph-mon[32448]: pgmap v107: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:36 smithi117 ceph-mon[36002]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:36 smithi120 ceph-mon[36826]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:36 smithi023 ceph-mon[32448]: pgmap v108: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:38 smithi120 ceph-mon[36826]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:38 smithi023 ceph-mon[32448]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:38 smithi117 ceph-mon[36002]: pgmap v109: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:40 smithi120 ceph-mon[36826]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:40 smithi023 ceph-mon[32448]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:40 smithi117 ceph-mon[36002]: pgmap v110: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:42 smithi120 ceph-mon[36826]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:42.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:42 smithi023 ceph-mon[32448]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:42 smithi117 ceph-mon[36002]: pgmap v111: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:44.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:44 smithi023 ceph-mon[32448]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:44 smithi117 ceph-mon[36002]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:44 smithi120 ceph-mon[36826]: pgmap v112: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:46 smithi023 ceph-mon[32448]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:46 smithi117 ceph-mon[36002]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:46 smithi120 ceph-mon[36826]: pgmap v113: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:48.639 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:48 smithi120 ceph-mon[36826]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:48 smithi023 ceph-mon[32448]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:48 smithi117 ceph-mon[36002]: pgmap v114: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:50.623 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:50 smithi117 ceph-mon[36002]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:50 smithi023 ceph-mon[32448]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:50 smithi120 ceph-mon[36826]: pgmap v115: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:52 smithi023 ceph-mon[32448]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:52 smithi117 ceph-mon[36002]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:52 smithi120 ceph-mon[36826]: pgmap v116: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:54 smithi023 ceph-mon[32448]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:54 smithi117 ceph-mon[36002]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:54 smithi120 ceph-mon[36826]: pgmap v117: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:56 smithi023 ceph-mon[32448]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:56 smithi117 ceph-mon[36002]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:56 smithi120 ceph-mon[36826]: pgmap v118: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:56:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:56:58 smithi023 ceph-mon[32448]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:56:58 smithi117 ceph-mon[36002]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:56:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:56:58 smithi120 ceph-mon[36826]: pgmap v119: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:00 smithi023 ceph-mon[32448]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:00 smithi117 ceph-mon[36002]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:00 smithi120 ceph-mon[36826]: pgmap v120: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:02 smithi023 ceph-mon[32448]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:02 smithi117 ceph-mon[36002]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:02.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:02 smithi120 ceph-mon[36826]: pgmap v121: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:04 smithi023 ceph-mon[32448]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:04 smithi117 ceph-mon[36002]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:04 smithi120 ceph-mon[36826]: pgmap v122: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:06 smithi023 ceph-mon[32448]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:06.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:06 smithi117 ceph-mon[36002]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:06 smithi120 ceph-mon[36826]: pgmap v123: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:08 smithi023 ceph-mon[32448]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:08 smithi117 ceph-mon[36002]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:08 smithi120 ceph-mon[36826]: pgmap v124: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:10 smithi023 ceph-mon[32448]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:10 smithi117 ceph-mon[36002]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:10 smithi120 ceph-mon[36826]: pgmap v125: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:57:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:12 smithi023 ceph-mon[32448]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:12 smithi117 ceph-mon[36002]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:12 smithi120 ceph-mon[36826]: pgmap v126: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:57:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:14 smithi023 ceph-mon[32448]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:14 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:57:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:14 smithi117 ceph-mon[36002]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:14 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:57:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:14 smithi120 ceph-mon[36826]: pgmap v127: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:57:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:57:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:57:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:14 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:57:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:16 smithi023 ceph-mon[32448]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:16 smithi117 ceph-mon[36002]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:16 smithi120 ceph-mon[36826]: pgmap v128: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:57:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:18 smithi117 ceph-mon[36002]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:57:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:18 smithi120 ceph-mon[36826]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:57:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:18 smithi023 ceph-mon[32448]: pgmap v129: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:57:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:20 smithi117 ceph-mon[36002]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:57:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:20 smithi120 ceph-mon[36826]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:57:20.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:20 smithi023 ceph-mon[32448]: pgmap v130: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:57:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:22 smithi117 ceph-mon[36002]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:57:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:22 smithi120 ceph-mon[36826]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:57:22.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:22 smithi023 ceph-mon[32448]: pgmap v131: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:57:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:24 smithi117 ceph-mon[36002]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:24 smithi120 ceph-mon[36826]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:24 smithi023 ceph-mon[32448]: pgmap v132: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:26 smithi117 ceph-mon[36002]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:26 smithi120 ceph-mon[36826]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:26.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:26 smithi023 ceph-mon[32448]: pgmap v133: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:28.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:28 smithi117 ceph-mon[36002]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:28.918 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:28 smithi120 ceph-mon[36826]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:28.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:28 smithi023 ceph-mon[32448]: pgmap v134: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:30.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:30 smithi023 ceph-mon[32448]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:30 smithi117 ceph-mon[36002]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:30 smithi120 ceph-mon[36826]: pgmap v135: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:32 smithi117 ceph-mon[36002]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:32 smithi120 ceph-mon[36826]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:32.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:32 smithi023 ceph-mon[32448]: pgmap v136: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:34 smithi117 ceph-mon[36002]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:34 smithi120 ceph-mon[36826]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:34 smithi023 ceph-mon[32448]: pgmap v137: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:36 smithi117 ceph-mon[36002]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:36 smithi120 ceph-mon[36826]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:36 smithi023 ceph-mon[32448]: pgmap v138: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:38 smithi117 ceph-mon[36002]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:38 smithi120 ceph-mon[36826]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:38.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:38 smithi023 ceph-mon[32448]: pgmap v139: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:40 smithi117 ceph-mon[36002]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:40 smithi120 ceph-mon[36826]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:40 smithi023 ceph-mon[32448]: pgmap v140: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:42 smithi117 ceph-mon[36002]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:42 smithi120 ceph-mon[36826]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:42 smithi023 ceph-mon[32448]: pgmap v141: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:44 smithi117 ceph-mon[36002]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:44 smithi120 ceph-mon[36826]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:44 smithi023 ceph-mon[32448]: pgmap v142: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:46 smithi117 ceph-mon[36002]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:46 smithi120 ceph-mon[36826]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:46 smithi023 ceph-mon[32448]: pgmap v143: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:48 smithi117 ceph-mon[36002]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:48 smithi120 ceph-mon[36826]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:48 smithi023 ceph-mon[32448]: pgmap v144: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:50 smithi117 ceph-mon[36002]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:50 smithi120 ceph-mon[36826]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:50.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:50 smithi023 ceph-mon[32448]: pgmap v145: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:52 smithi117 ceph-mon[36002]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:52 smithi120 ceph-mon[36826]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:52 smithi023 ceph-mon[32448]: pgmap v146: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:54 smithi117 ceph-mon[36002]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:54 smithi120 ceph-mon[36826]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:54 smithi023 ceph-mon[32448]: pgmap v147: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:56 smithi117 ceph-mon[36002]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:56 smithi120 ceph-mon[36826]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:56 smithi023 ceph-mon[32448]: pgmap v148: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:57:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:57:58 smithi117 ceph-mon[36002]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:57:58 smithi120 ceph-mon[36826]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:57:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:57:58 smithi023 ceph-mon[32448]: pgmap v149: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:00 smithi117 ceph-mon[36002]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:00 smithi120 ceph-mon[36826]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:00.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:00 smithi023 ceph-mon[32448]: pgmap v150: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:02 smithi117 ceph-mon[36002]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:02 smithi120 ceph-mon[36826]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:02 smithi023 ceph-mon[32448]: pgmap v151: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:04 smithi117 ceph-mon[36002]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:04 smithi120 ceph-mon[36826]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:04.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:04 smithi023 ceph-mon[32448]: pgmap v152: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:06 smithi117 ceph-mon[36002]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:06 smithi120 ceph-mon[36826]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:06 smithi023 ceph-mon[32448]: pgmap v153: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:08 smithi117 ceph-mon[36002]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:08 smithi120 ceph-mon[36826]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:08.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:08 smithi023 ceph-mon[32448]: pgmap v154: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:10 smithi117 ceph-mon[36002]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:10 smithi120 ceph-mon[36826]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:10.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:10 smithi023 ceph-mon[32448]: pgmap v155: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:58:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:12 smithi117 ceph-mon[36002]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:12 smithi120 ceph-mon[36826]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:12 smithi023 ceph-mon[32448]: pgmap v156: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:58:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:14 smithi117 ceph-mon[36002]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:14 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:14.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:14 smithi120 ceph-mon[36826]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:14.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:14 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:14 smithi023 ceph-mon[32448]: pgmap v157: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:14 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:58:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:15 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:15 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:15 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:58:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:15 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:15 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:15 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:58:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:15 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:58:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:15 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:58:15.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:15 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:58:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:16 smithi117 ceph-mon[36002]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:16 smithi120 ceph-mon[36826]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:16.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:16 smithi023 ceph-mon[32448]: pgmap v158: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:58:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:18 smithi117 ceph-mon[36002]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:58:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:18 smithi120 ceph-mon[36826]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:58:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:18 smithi023 ceph-mon[32448]: pgmap v159: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:58:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:20 smithi120 ceph-mon[36826]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:58:20.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:20 smithi023 ceph-mon[32448]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:58:21.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:20 smithi117 ceph-mon[36002]: pgmap v160: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:58:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:22 smithi120 ceph-mon[36826]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:58:22.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:22 smithi023 ceph-mon[32448]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:58:23.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:22 smithi117 ceph-mon[36002]: pgmap v161: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:58:23.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:23 smithi023 ceph-mon[32448]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:24.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:23 smithi117 ceph-mon[36002]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:24.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:23 smithi120 ceph-mon[36826]: pgmap v162: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:26.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:26 smithi117 ceph-mon[36002]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:26.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:26 smithi120 ceph-mon[36826]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:26.672 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:26 smithi023 ceph-mon[32448]: pgmap v163: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:28.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:28 smithi117 ceph-mon[36002]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:28.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:28 smithi120 ceph-mon[36826]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:28 smithi023 ceph-mon[32448]: pgmap v164: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:30.573 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:30 smithi117 ceph-mon[36002]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:30.597 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:30 smithi120 ceph-mon[36826]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:30.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:30 smithi023 ceph-mon[32448]: pgmap v165: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:32 smithi117 ceph-mon[36002]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:32 smithi120 ceph-mon[36826]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:32.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:32 smithi023 ceph-mon[32448]: pgmap v166: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:34 smithi117 ceph-mon[36002]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:34.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:34 smithi120 ceph-mon[36826]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:34 smithi023 ceph-mon[32448]: pgmap v167: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:36.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:36 smithi117 ceph-mon[36002]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:36.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:36 smithi120 ceph-mon[36826]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:36 smithi023 ceph-mon[32448]: pgmap v168: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:38.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:38 smithi117 ceph-mon[36002]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:38.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:38 smithi120 ceph-mon[36826]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:38 smithi023 ceph-mon[32448]: pgmap v169: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:40.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:40 smithi117 ceph-mon[36002]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:40.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:40 smithi120 ceph-mon[36826]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:40 smithi023 ceph-mon[32448]: pgmap v170: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:42.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:42 smithi117 ceph-mon[36002]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:42.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:42 smithi120 ceph-mon[36826]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:42.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:42 smithi023 ceph-mon[32448]: pgmap v171: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:44 smithi120 ceph-mon[36826]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:44.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:44 smithi023 ceph-mon[32448]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:44 smithi117 ceph-mon[36002]: pgmap v172: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:46.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:46 smithi120 ceph-mon[36826]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:46 smithi023 ceph-mon[32448]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:46 smithi117 ceph-mon[36002]: pgmap v173: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:48.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:48 smithi120 ceph-mon[36826]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:48.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:48 smithi023 ceph-mon[32448]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:48 smithi117 ceph-mon[36002]: pgmap v174: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:50.590 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:50 smithi120 ceph-mon[36826]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:50.623 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:50 smithi117 ceph-mon[36002]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:50 smithi023 ceph-mon[32448]: pgmap v175: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:52 smithi120 ceph-mon[36826]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:52 smithi023 ceph-mon[32448]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:52 smithi117 ceph-mon[36002]: pgmap v176: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:54.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:54 smithi023 ceph-mon[32448]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:54 smithi117 ceph-mon[36002]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:54 smithi120 ceph-mon[36826]: pgmap v177: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:56 smithi023 ceph-mon[32448]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:56 smithi117 ceph-mon[36002]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:56 smithi120 ceph-mon[36826]: pgmap v178: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:58:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:58:58 smithi023 ceph-mon[32448]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:58:58 smithi117 ceph-mon[36002]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:58:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:58:58 smithi120 ceph-mon[36826]: pgmap v179: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:00 smithi023 ceph-mon[32448]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:00 smithi117 ceph-mon[36002]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:00 smithi120 ceph-mon[36826]: pgmap v180: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:02 smithi023 ceph-mon[32448]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:02 smithi117 ceph-mon[36002]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:02 smithi120 ceph-mon[36826]: pgmap v181: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:04.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:04 smithi023 ceph-mon[32448]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:04.824 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:04 smithi117 ceph-mon[36002]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:04.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:04 smithi120 ceph-mon[36826]: pgmap v182: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:06 smithi023 ceph-mon[32448]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:06 smithi117 ceph-mon[36002]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:06 smithi120 ceph-mon[36826]: pgmap v183: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:08.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:08 smithi023 ceph-mon[32448]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:08 smithi117 ceph-mon[36002]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:08 smithi120 ceph-mon[36826]: pgmap v184: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:10 smithi023 ceph-mon[32448]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:10 smithi117 ceph-mon[36002]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:10 smithi120 ceph-mon[36826]: pgmap v185: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 23 op/s 2024-03-28T15:59:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:12 smithi023 ceph-mon[32448]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:12 smithi117 ceph-mon[36002]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:12 smithi120 ceph-mon[36826]: pgmap v186: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 22 KiB/s wr, 27 op/s 2024-03-28T15:59:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:14 smithi023 ceph-mon[32448]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:14 smithi117 ceph-mon[36002]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:14 smithi120 ceph-mon[36826]: pgmap v187: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:15.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:15 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:15 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:15 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T15:59:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:16 smithi023 ceph-mon[32448]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:16 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:16 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:16 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:59:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:16 smithi117 ceph-mon[36002]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:16 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:16 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:16 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:59:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:16 smithi120 ceph-mon[36826]: pgmap v188: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 170 B/s wr, 8 op/s 2024-03-28T15:59:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:16 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T15:59:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:16 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T15:59:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:16 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T15:59:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:18 smithi023 ceph-mon[32448]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:59:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:18 smithi117 ceph-mon[36002]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:59:18.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:18 smithi120 ceph-mon[36826]: pgmap v189: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 170 B/s wr, 12 op/s 2024-03-28T15:59:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:20 smithi023 ceph-mon[32448]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:59:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:20 smithi117 ceph-mon[36002]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:59:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:20 smithi120 ceph-mon[36826]: pgmap v190: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-28T15:59:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:22 smithi023 ceph-mon[32448]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:59:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:22 smithi117 ceph-mon[36002]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:59:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:22 smithi120 ceph-mon[36826]: pgmap v191: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T15:59:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:24 smithi023 ceph-mon[32448]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:24 smithi117 ceph-mon[36002]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:24 smithi120 ceph-mon[36826]: pgmap v192: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:26 smithi023 ceph-mon[32448]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:26 smithi117 ceph-mon[36002]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:26 smithi120 ceph-mon[36826]: pgmap v193: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:28 smithi023 ceph-mon[32448]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:28 smithi117 ceph-mon[36002]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:28.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:28 smithi120 ceph-mon[36826]: pgmap v194: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:30 smithi023 ceph-mon[32448]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:30 smithi117 ceph-mon[36002]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:30 smithi120 ceph-mon[36826]: pgmap v195: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:32 smithi023 ceph-mon[32448]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:32 smithi117 ceph-mon[36002]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:32 smithi120 ceph-mon[36826]: pgmap v196: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:34 smithi117 ceph-mon[36002]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:34 smithi120 ceph-mon[36826]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:34 smithi023 ceph-mon[32448]: pgmap v197: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:36 smithi117 ceph-mon[36002]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:36.844 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:36 smithi120 ceph-mon[36826]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:36.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:36 smithi023 ceph-mon[32448]: pgmap v198: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:38 smithi117 ceph-mon[36002]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:38 smithi120 ceph-mon[36826]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:38.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:38 smithi023 ceph-mon[32448]: pgmap v199: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:40 smithi117 ceph-mon[36002]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:40 smithi120 ceph-mon[36826]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:40 smithi023 ceph-mon[32448]: pgmap v200: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:42 smithi117 ceph-mon[36002]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:42 smithi120 ceph-mon[36826]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:42 smithi023 ceph-mon[32448]: pgmap v201: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:44 smithi117 ceph-mon[36002]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:44 smithi120 ceph-mon[36826]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:44 smithi023 ceph-mon[32448]: pgmap v202: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:46 smithi117 ceph-mon[36002]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:46 smithi120 ceph-mon[36826]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:46 smithi023 ceph-mon[32448]: pgmap v203: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:48 smithi117 ceph-mon[36002]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:48 smithi120 ceph-mon[36826]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:48 smithi023 ceph-mon[32448]: pgmap v204: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:50 smithi117 ceph-mon[36002]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:50 smithi120 ceph-mon[36826]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:50.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:50 smithi023 ceph-mon[32448]: pgmap v205: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:52 smithi117 ceph-mon[36002]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:52 smithi120 ceph-mon[36826]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:52 smithi023 ceph-mon[32448]: pgmap v206: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:54 smithi117 ceph-mon[36002]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:54 smithi120 ceph-mon[36826]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:54 smithi023 ceph-mon[32448]: pgmap v207: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:56 smithi117 ceph-mon[36002]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:56 smithi120 ceph-mon[36826]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:56 smithi023 ceph-mon[32448]: pgmap v208: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T15:59:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 15:59:58 smithi117 ceph-mon[36002]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 15:59:58 smithi120 ceph-mon[36826]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T15:59:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 15:59:58 smithi023 ceph-mon[32448]: pgmap v209: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:00.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:00 smithi023 ceph-mon[32448]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:00.723 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:00 smithi023 ceph-mon[32448]: overall HEALTH_OK 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:00 smithi117 ceph-mon[36002]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:00 smithi117 ceph-mon[36002]: overall HEALTH_OK 2024-03-28T16:00:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:00 smithi120 ceph-mon[36826]: pgmap v210: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:00.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:00 smithi120 ceph-mon[36826]: overall HEALTH_OK 2024-03-28T16:00:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:02 smithi117 ceph-mon[36002]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:02 smithi120 ceph-mon[36826]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:02 smithi023 ceph-mon[32448]: pgmap v211: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:04 smithi117 ceph-mon[36002]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:04 smithi120 ceph-mon[36826]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:04.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:04 smithi023 ceph-mon[32448]: pgmap v212: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:06 smithi117 ceph-mon[36002]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:06 smithi120 ceph-mon[36826]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:06 smithi023 ceph-mon[32448]: pgmap v213: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:00:08.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:08 smithi117 ceph-mon[36002]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:08 smithi120 ceph-mon[36826]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:08.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:08 smithi023 ceph-mon[32448]: pgmap v214: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:00:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:09 smithi117 ceph-mon[36002]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T16:00:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:09 smithi120 ceph-mon[36826]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T16:00:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:09 smithi023 ceph-mon[32448]: osdmap e96: 12 total, 12 up, 12 in 2024-03-28T16:00:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:10 smithi117 ceph-mon[36002]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:00:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:10 smithi120 ceph-mon[36826]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:00:10.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:10 smithi023 ceph-mon[32448]: pgmap v216: 129 pgs: 129 active+clean; 259 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:00:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:12 smithi117 ceph-mon[36002]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:12 smithi120 ceph-mon[36826]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:12 smithi023 ceph-mon[32448]: pgmap v217: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:14 smithi117 ceph-mon[36002]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:14 smithi117 ceph-mon[36002]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T16:00:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:14 smithi120 ceph-mon[36826]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:14 smithi120 ceph-mon[36826]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T16:00:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:14 smithi023 ceph-mon[32448]: pgmap v218: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:14 smithi023 ceph-mon[32448]: osdmap e97: 12 total, 12 up, 12 in 2024-03-28T16:00:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:16 smithi117 ceph-mon[36002]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:00:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:16 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:16 smithi120 ceph-mon[36826]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:00:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:16 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:16.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:16 smithi023 ceph-mon[32448]: pgmap v220: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:00:16.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:16 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:00:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:17 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:17 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:17 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:00:17.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:17 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:17.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:17 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:17.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:17 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:00:17.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:17 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:00:17.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:17 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:00:17.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:17 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:00:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:18 smithi117 ceph-mon[36002]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 234 B/s wr, 11 op/s 2024-03-28T16:00:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:18 smithi120 ceph-mon[36826]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 234 B/s wr, 11 op/s 2024-03-28T16:00:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:18 smithi023 ceph-mon[32448]: pgmap v221: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 234 B/s wr, 11 op/s 2024-03-28T16:00:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:20 smithi117 ceph-mon[36002]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:20 smithi120 ceph-mon[36826]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:20.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:20 smithi023 ceph-mon[32448]: pgmap v222: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:00:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:22 smithi117 ceph-mon[36002]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:22 smithi120 ceph-mon[36826]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:22.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:22 smithi023 ceph-mon[32448]: pgmap v223: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:24 smithi117 ceph-mon[36002]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:24 smithi120 ceph-mon[36826]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:24 smithi023 ceph-mon[32448]: pgmap v224: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:00:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:26 smithi117 ceph-mon[36002]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:00:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:26 smithi120 ceph-mon[36826]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:00:26.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:26 smithi023 ceph-mon[32448]: pgmap v225: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:00:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:28 smithi117 ceph-mon[36002]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:00:28.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:28 smithi120 ceph-mon[36826]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:00:28.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:28 smithi023 ceph-mon[32448]: pgmap v226: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:00:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:30 smithi117 ceph-mon[36002]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:30 smithi120 ceph-mon[36826]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:30.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:30 smithi023 ceph-mon[32448]: pgmap v227: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:32 smithi117 ceph-mon[36002]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:32 smithi120 ceph-mon[36826]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:32.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:32 smithi023 ceph-mon[32448]: pgmap v228: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:34 smithi117 ceph-mon[36002]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:34 smithi120 ceph-mon[36826]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:34 smithi023 ceph-mon[32448]: pgmap v229: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:36 smithi117 ceph-mon[36002]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:36 smithi120 ceph-mon[36826]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:36.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:36 smithi023 ceph-mon[32448]: pgmap v230: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:38 smithi117 ceph-mon[36002]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:38 smithi120 ceph-mon[36826]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:38.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:38 smithi023 ceph-mon[32448]: pgmap v231: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:40 smithi120 ceph-mon[36826]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:40 smithi023 ceph-mon[32448]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:41.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:40 smithi117 ceph-mon[36002]: pgmap v232: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:42 smithi120 ceph-mon[36826]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:42 smithi023 ceph-mon[32448]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:43.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:42 smithi117 ceph-mon[36002]: pgmap v233: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:44 smithi120 ceph-mon[36826]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:44 smithi023 ceph-mon[32448]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:45.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:44 smithi117 ceph-mon[36002]: pgmap v234: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:46 smithi023 ceph-mon[32448]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:47.074 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:46 smithi117 ceph-mon[36002]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:47.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:46 smithi120 ceph-mon[36826]: pgmap v235: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:47.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:47 smithi023 ceph-mon[32448]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:48.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:47 smithi117 ceph-mon[36002]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:48.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:47 smithi120 ceph-mon[36826]: pgmap v236: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:50.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:50 smithi117 ceph-mon[36002]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:50.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:50 smithi120 ceph-mon[36826]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:50.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:50 smithi023 ceph-mon[32448]: pgmap v237: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:52.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:52 smithi117 ceph-mon[36002]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:52.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:52 smithi120 ceph-mon[36826]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:52.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:52 smithi023 ceph-mon[32448]: pgmap v238: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:54.575 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:54 smithi117 ceph-mon[36002]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:54.595 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:54 smithi120 ceph-mon[36826]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:54.673 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:54 smithi023 ceph-mon[32448]: pgmap v239: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:56 smithi117 ceph-mon[36002]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:56.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:56 smithi120 ceph-mon[36826]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:56.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:56 smithi023 ceph-mon[32448]: pgmap v240: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:00:58.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:00:58 smithi117 ceph-mon[36002]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:58.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:00:58 smithi120 ceph-mon[36826]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:00:58.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:00:58 smithi023 ceph-mon[32448]: pgmap v241: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:00.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:00 smithi023 ceph-mon[32448]: pgmap v242: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:00 smithi117 ceph-mon[36002]: pgmap v242: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:00 smithi120 ceph-mon[36826]: pgmap v242: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:02.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:02 smithi023 ceph-mon[32448]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:02 smithi117 ceph-mon[36002]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:02 smithi120 ceph-mon[36826]: pgmap v243: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:04.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:04 smithi023 ceph-mon[32448]: pgmap v244: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:04 smithi117 ceph-mon[36002]: pgmap v244: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:04 smithi120 ceph-mon[36826]: pgmap v244: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:06 smithi023 ceph-mon[32448]: pgmap v245: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:06 smithi117 ceph-mon[36002]: pgmap v245: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:06 smithi120 ceph-mon[36826]: pgmap v245: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:01:08.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:08 smithi023 ceph-mon[32448]: pgmap v246: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:08 smithi117 ceph-mon[36002]: pgmap v246: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:08 smithi120 ceph-mon[36826]: pgmap v246: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:01:09.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:09 smithi023 ceph-mon[32448]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T16:01:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:09 smithi117 ceph-mon[36002]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T16:01:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:09 smithi120 ceph-mon[36826]: osdmap e98: 12 total, 12 up, 12 in 2024-03-28T16:01:10.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:10 smithi023 ceph-mon[32448]: pgmap v248: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:01:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:10 smithi117 ceph-mon[36002]: pgmap v248: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:01:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:10 smithi120 ceph-mon[36826]: pgmap v248: 129 pgs: 129 active+clean; 259 MiB data, 544 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:01:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:12 smithi023 ceph-mon[32448]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:12 smithi117 ceph-mon[36002]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:12 smithi120 ceph-mon[36826]: pgmap v249: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:14 smithi023 ceph-mon[32448]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:14.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:14 smithi023 ceph-mon[32448]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T16:01:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:14 smithi117 ceph-mon[36002]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:14 smithi117 ceph-mon[36002]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T16:01:14.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:14 smithi120 ceph-mon[36826]: pgmap v250: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:14.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:14 smithi120 ceph-mon[36826]: osdmap e99: 12 total, 12 up, 12 in 2024-03-28T16:01:16.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:16 smithi023 ceph-mon[32448]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:01:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:16 smithi117 ceph-mon[36002]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:01:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:16 smithi120 ceph-mon[36826]: pgmap v252: 129 pgs: 129 active+clean; 2.3 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:01:17.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:17 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:17.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:17 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:17.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:17 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:01:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:18 smithi023 ceph-mon[32448]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-28T16:01:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:18 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:18 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:18 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:01:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:18 smithi117 ceph-mon[36002]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-28T16:01:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:18 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:18.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:18 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:18.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:18 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:01:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:18 smithi120 ceph-mon[36826]: pgmap v253: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 230 B/s wr, 11 op/s 2024-03-28T16:01:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:18 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:01:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:18 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:01:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:18 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:01:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:20 smithi023 ceph-mon[32448]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:20 smithi117 ceph-mon[36002]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:20 smithi120 ceph-mon[36826]: pgmap v254: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:01:22.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:22 smithi023 ceph-mon[32448]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:22 smithi117 ceph-mon[36002]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:22 smithi120 ceph-mon[36826]: pgmap v255: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:24.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:24 smithi023 ceph-mon[32448]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:24 smithi117 ceph-mon[36002]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:24 smithi120 ceph-mon[36826]: pgmap v256: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:01:26.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:26 smithi023 ceph-mon[32448]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:01:26.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:26 smithi117 ceph-mon[36002]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:01:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:26 smithi120 ceph-mon[36826]: pgmap v257: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:01:28.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:28 smithi023 ceph-mon[32448]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:01:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:28 smithi117 ceph-mon[36002]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:01:28.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:28 smithi120 ceph-mon[36826]: pgmap v258: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:01:30.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:30 smithi023 ceph-mon[32448]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:30 smithi117 ceph-mon[36002]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:30 smithi120 ceph-mon[36826]: pgmap v259: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:32 smithi023 ceph-mon[32448]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:32 smithi117 ceph-mon[36002]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:32 smithi120 ceph-mon[36826]: pgmap v260: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:34 smithi023 ceph-mon[32448]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:34 smithi117 ceph-mon[36002]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:34 smithi120 ceph-mon[36826]: pgmap v261: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:36 smithi023 ceph-mon[32448]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:36 smithi117 ceph-mon[36002]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:36 smithi120 ceph-mon[36826]: pgmap v262: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:38 smithi023 ceph-mon[32448]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:38 smithi117 ceph-mon[36002]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:38 smithi120 ceph-mon[36826]: pgmap v263: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:40 smithi117 ceph-mon[36002]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:40 smithi120 ceph-mon[36826]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:40 smithi023 ceph-mon[32448]: pgmap v264: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:42 smithi117 ceph-mon[36002]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:42 smithi120 ceph-mon[36826]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:42 smithi023 ceph-mon[32448]: pgmap v265: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:44 smithi117 ceph-mon[36002]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:44 smithi120 ceph-mon[36826]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:44 smithi023 ceph-mon[32448]: pgmap v266: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:46 smithi117 ceph-mon[36002]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:46 smithi120 ceph-mon[36826]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:46 smithi023 ceph-mon[32448]: pgmap v267: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:48 smithi117 ceph-mon[36002]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:48 smithi120 ceph-mon[36826]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:48 smithi023 ceph-mon[32448]: pgmap v268: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:50.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:50 smithi117 ceph-mon[36002]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:50 smithi120 ceph-mon[36826]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:50.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:50 smithi023 ceph-mon[32448]: pgmap v269: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:52 smithi117 ceph-mon[36002]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:52 smithi120 ceph-mon[36826]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:52 smithi023 ceph-mon[32448]: pgmap v270: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:54 smithi117 ceph-mon[36002]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:54 smithi120 ceph-mon[36826]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:54 smithi023 ceph-mon[32448]: pgmap v271: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:56.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:56 smithi117 ceph-mon[36002]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:56.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:56 smithi120 ceph-mon[36826]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:56 smithi023 ceph-mon[32448]: pgmap v272: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:01:58.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:01:58 smithi117 ceph-mon[36002]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:01:58 smithi120 ceph-mon[36826]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:01:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:01:58 smithi023 ceph-mon[32448]: pgmap v273: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:00.721 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:00 smithi023 ceph-mon[32448]: pgmap v274: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:00.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:00 smithi117 ceph-mon[36002]: pgmap v274: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:00.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:00 smithi120 ceph-mon[36826]: pgmap v274: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:02.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:02 smithi117 ceph-mon[36002]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:02.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:02 smithi120 ceph-mon[36826]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:02 smithi023 ceph-mon[32448]: pgmap v275: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:04.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:04 smithi117 ceph-mon[36002]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:04.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:04 smithi120 ceph-mon[36826]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:04.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:04 smithi023 ceph-mon[32448]: pgmap v276: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:06 smithi117 ceph-mon[36002]: pgmap v277: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:06.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:06 smithi120 ceph-mon[36826]: pgmap v277: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:06.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:06 smithi023 ceph-mon[32448]: pgmap v277: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:02:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:08 smithi117 ceph-mon[36002]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:08 smithi120 ceph-mon[36826]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:08.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:08 smithi023 ceph-mon[32448]: pgmap v278: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:02:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:09 smithi117 ceph-mon[36002]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T16:02:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:09 smithi120 ceph-mon[36826]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T16:02:09.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:09 smithi023 ceph-mon[32448]: osdmap e100: 12 total, 12 up, 12 in 2024-03-28T16:02:10.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:10 smithi117 ceph-mon[36002]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:02:10.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:10 smithi120 ceph-mon[36826]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:02:10.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:10 smithi023 ceph-mon[32448]: pgmap v280: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:02:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:12 smithi117 ceph-mon[36002]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:12 smithi120 ceph-mon[36826]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:12.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:12 smithi023 ceph-mon[32448]: pgmap v281: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:14 smithi117 ceph-mon[36002]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:14 smithi117 ceph-mon[36002]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T16:02:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:14 smithi120 ceph-mon[36826]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:14.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:14 smithi120 ceph-mon[36826]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T16:02:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:14 smithi023 ceph-mon[32448]: pgmap v282: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:14.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:14 smithi023 ceph-mon[32448]: osdmap e101: 12 total, 12 up, 12 in 2024-03-28T16:02:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:16 smithi117 ceph-mon[36002]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:02:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:16 smithi120 ceph-mon[36826]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:02:16.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:16 smithi023 ceph-mon[32448]: pgmap v284: 129 pgs: 129 active+clean; 2.3 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 127 B/s wr, 6 op/s 2024-03-28T16:02:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:18 smithi117 ceph-mon[36002]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2024-03-28T16:02:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:18 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:18 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:18 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:18.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:18 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:02:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:18 smithi120 ceph-mon[36826]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2024-03-28T16:02:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:18 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:18 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:18 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:18 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:02:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:18 smithi023 ceph-mon[32448]: pgmap v285: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 233 B/s wr, 11 op/s 2024-03-28T16:02:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:18 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:02:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:18 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:02:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:18 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:02:18.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:18 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:02:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:20 smithi117 ceph-mon[36002]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:20 smithi120 ceph-mon[36826]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:20.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:20 smithi023 ceph-mon[32448]: pgmap v286: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 204 B/s wr, 10 op/s 2024-03-28T16:02:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:22 smithi117 ceph-mon[36002]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:22 smithi120 ceph-mon[36826]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:22.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:22 smithi023 ceph-mon[32448]: pgmap v287: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:24.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:24 smithi117 ceph-mon[36002]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:24.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:24 smithi120 ceph-mon[36826]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:24 smithi023 ceph-mon[32448]: pgmap v288: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:02:26.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:26 smithi117 ceph-mon[36002]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:02:26.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:26 smithi120 ceph-mon[36826]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:02:26.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:26 smithi023 ceph-mon[32448]: pgmap v289: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:02:28.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:28 smithi117 ceph-mon[36002]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:02:28.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:28 smithi120 ceph-mon[36826]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:02:28.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:28 smithi023 ceph-mon[32448]: pgmap v290: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:02:30.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:30 smithi117 ceph-mon[36002]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:30.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:30 smithi120 ceph-mon[36826]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:30.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:30 smithi023 ceph-mon[32448]: pgmap v291: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:32.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:32 smithi117 ceph-mon[36002]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:32.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:32 smithi120 ceph-mon[36826]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:32.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:32 smithi023 ceph-mon[32448]: pgmap v292: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:34.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:34 smithi117 ceph-mon[36002]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:34 smithi120 ceph-mon[36826]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:34.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:34 smithi023 ceph-mon[32448]: pgmap v293: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:36 smithi117 ceph-mon[36002]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:36.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:36 smithi120 ceph-mon[36826]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:36.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:36 smithi023 ceph-mon[32448]: pgmap v294: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:38 smithi117 ceph-mon[36002]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:38.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:38 smithi120 ceph-mon[36826]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:38.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:38 smithi023 ceph-mon[32448]: pgmap v295: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:40 smithi117 ceph-mon[36002]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:40.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:40 smithi120 ceph-mon[36826]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:40.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:40 smithi023 ceph-mon[32448]: pgmap v296: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:42 smithi117 ceph-mon[36002]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:42.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:42 smithi120 ceph-mon[36826]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:42.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:42 smithi023 ceph-mon[32448]: pgmap v297: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:44 smithi117 ceph-mon[36002]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:44.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:44 smithi120 ceph-mon[36826]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:44.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:44 smithi023 ceph-mon[32448]: pgmap v298: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:46 smithi120 ceph-mon[36826]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:46.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:46 smithi023 ceph-mon[32448]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:47.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:46 smithi117 ceph-mon[36002]: pgmap v299: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:48 smithi120 ceph-mon[36826]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:48.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:48 smithi023 ceph-mon[32448]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:49.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:48 smithi117 ceph-mon[36002]: pgmap v300: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:50.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:50 smithi120 ceph-mon[36826]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:50.873 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:50 smithi117 ceph-mon[36002]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:50.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:50 smithi023 ceph-mon[32448]: pgmap v301: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:52 smithi120 ceph-mon[36826]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:52 smithi023 ceph-mon[32448]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:53.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:52 smithi117 ceph-mon[36002]: pgmap v302: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:54 smithi023 ceph-mon[32448]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:55.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:54 smithi117 ceph-mon[36002]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:55.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:54 smithi120 ceph-mon[36826]: pgmap v303: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:56.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:56 smithi023 ceph-mon[32448]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:57.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:56 smithi117 ceph-mon[36002]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:57.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:56 smithi120 ceph-mon[36826]: pgmap v304: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:02:58.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:02:58 smithi023 ceph-mon[32448]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:59.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:02:58 smithi117 ceph-mon[36002]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:02:59.152 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:02:58 smithi120 ceph-mon[36826]: pgmap v305: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:00.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:00 smithi023 ceph-mon[32448]: pgmap v306: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:01.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:00 smithi117 ceph-mon[36002]: pgmap v306: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:01.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:00 smithi120 ceph-mon[36826]: pgmap v306: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:02.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:02 smithi023 ceph-mon[32448]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:03.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:02 smithi117 ceph-mon[36002]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:03.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:02 smithi120 ceph-mon[36826]: pgmap v307: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:03.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:03 smithi023 ceph-mon[32448]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:04.071 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:03 smithi117 ceph-mon[36002]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:04.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:03 smithi120 ceph-mon[36826]: pgmap v308: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:06.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:06 smithi120 ceph-mon[36826]: pgmap v309: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:06.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:06 smithi023 ceph-mon[32448]: pgmap v309: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:06.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:06 smithi117 ceph-mon[36002]: pgmap v309: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 25 op/s 2024-03-28T16:03:07.206 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-28T16:03:07.337 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- 172.21.15.23:0/2739073079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 msgr2=0x7f0ea410d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 --2- 172.21.15.23:0/2739073079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea410d390 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c009cd0 tx=0x7f0e9c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- 172.21.15.23:0/2739073079 shutdown_connections 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 --2- 172.21.15.23:0/2739073079 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea410d390 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 --2- 172.21.15.23:0/2739073079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4100880 0x7f0ea4100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 --2- 172.21.15.23:0/2739073079 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4106730 0x7f0ea4106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- 172.21.15.23:0/2739073079 >> 172.21.15.23:0/2739073079 conn(0x7f0ea4077580 msgr2=0x7f0ea4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- 172.21.15.23:0/2739073079 shutdown_connections 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- 172.21.15.23:0/2739073079 wait complete. 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 Processor -- start 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.335+0000 7f0ea9875640 1 -- start start 2024-03-28T16:03:07.338 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 0x7f0ea419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4106730 0x7f0ea419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea41109c0 con 0x7f0ea4101240 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4110840 con 0x7f0ea4106730 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea9875640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ea4110b40 con 0x7f0ea4100880 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39370/0 (socket says 172.21.15.23:39370) 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea2ffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 0x7f0ea419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 -- 172.21.15.23:0/229181039 learned_addr learned my addr 172.21.15.23:0/229181039 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea37fe640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4106730 0x7f0ea419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 -- 172.21.15.23:0/229181039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 msgr2=0x7f0ea419a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 0x7f0ea419a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.339 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 -- 172.21.15.23:0/229181039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4106730 msgr2=0x7f0ea419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4106730 0x7f0ea419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 -- 172.21.15.23:0/229181039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea419f040 con 0x7f0ea4101240 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea2ffd640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 0x7f0ea419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea27fc640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800e9f0 tx=0x7f0e9800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.336+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9800cd80 con 0x7f0ea4101240 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.337+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e98010910 con 0x7f0ea4101240 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.337+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea419f330 con 0x7f0ea4101240 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.337+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e98018990 con 0x7f0ea4101240 2024-03-28T16:03:07.340 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.337+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea41a6bc0 con 0x7f0ea4101240 2024-03-28T16:03:07.341 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.338+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea4101ca0 con 0x7f0ea4101240 2024-03-28T16:03:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.339+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0e98010ac0 con 0x7f0ea4101240 2024-03-28T16:03:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.339+0000 7f0ea8873640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 0x7f0e8c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.339+0000 7f0ea2ffd640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 0x7f0e8c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.339+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f0e980cf520 con 0x7f0ea4101240 2024-03-28T16:03:07.342 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.340+0000 7f0ea2ffd640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 0x7f0e8c0a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009fc0 tx=0x7f0e94005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:07.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.346+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0e9809b810 con 0x7f0ea4101240 2024-03-28T16:03:07.553 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.553+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ea40ffe80 con 0x7f0e8c0a2a90 2024-03-28T16:03:07.560 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.560+0000 7f0ea8873640 1 -- 172.21.15.23:0/229181039 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f0ea40ffe80 con 0x7f0e8c0a2a90 2024-03-28T16:03:07.561 INFO:teuthology.orchestra.run.smithi023.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-28T00:00:00", "created": "2024-03-28T15:53:06", "first": "2024-03-28T15:54:00", "last": "2024-03-28T16:03:00", "last_pruned": "2024-03-28T16:03:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-28T16:03:07.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 msgr2=0x7f0e8c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 0x7f0e8c0a4f50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0e94009fc0 tx=0x7f0e94005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:07.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 msgr2=0x7f0ea419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e9800e9f0 tx=0x7f0e9800eeb0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.563 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 shutdown_connections 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0e8c0a2a90 0x7f0e8c0a4f50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ea4106730 0x7f0ea419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0ea4101240 0x7f0ea419a590 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 --2- 172.21.15.23:0/229181039 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0ea4100880 0x7f0ea419a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 >> 172.21.15.23:0/229181039 conn(0x7f0ea4077580 msgr2=0x7f0ea40ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 shutdown_connections 2024-03-28T16:03:07.564 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.562+0000 7f0ea9875640 1 -- 172.21.15.23:0/229181039 wait complete. 2024-03-28T16:03:07.573 DEBUG:teuthology.orchestra.run.smithi023:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-28T16:03:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- 172.21.15.23:0/643465439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1011b0 msgr2=0x7fa3cc10d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/643465439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc10d3d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009c10 tx=0x7fa3c402bef0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.680 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- 172.21.15.23:0/643465439 shutdown_connections 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/643465439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc10d3d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/643465439 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/643465439 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 0x7fa3cc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- 172.21.15.23:0/643465439 >> 172.21.15.23:0/643465439 conn(0x7fa3cc0773f0 msgr2=0x7fa3cc077800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- 172.21.15.23:0/643465439 shutdown_connections 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- 172.21.15.23:0/643465439 wait complete. 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 Processor -- start 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.678+0000 7fa3d27f4640 1 -- start start 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 0x7fa3cc1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc110a60 con 0x7fa3cc1011b0 2024-03-28T16:03:07.681 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc1108e0 con 0x7fa3cc1007f0 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d27f4640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc110be0 con 0x7fa3cc105540 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cbfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39396/0 (socket says 172.21.15.23:39396) 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 -- 172.21.15.23:0/1816240428 learned_addr learned my addr 172.21.15.23:0/1816240428 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3d0d91640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 0x7fa3cc1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 -- 172.21.15.23:0/1816240428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 msgr2=0x7fa3cc1975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.682 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 0x7fa3cc1975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.683 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 -- 172.21.15.23:0/1816240428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 msgr2=0x7fa3cc196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 -- 172.21.15.23:0/1816240428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3cc1a63a0 con 0x7fa3cc1011b0 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cbfff640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3cb7fe640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ea60 tx=0x7fa3c000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.679+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c000cd20 con 0x7fa3cc1011b0 2024-03-28T16:03:07.686 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.680+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3c00035f0 con 0x7fa3cc1011b0 2024-03-28T16:03:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.680+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc1a6690 con 0x7fa3cc1011b0 2024-03-28T16:03:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.680+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0010690 con 0x7fa3cc1011b0 2024-03-28T16:03:07.687 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.680+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc1a6bb0 con 0x7fa3cc1011b0 2024-03-28T16:03:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.681+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc04f400 con 0x7fa3cc1011b0 2024-03-28T16:03:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.682+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa3c00037a0 con 0x7fa3cc1011b0 2024-03-28T16:03:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.682+0000 7fa3c97fa640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 0x7fa3b40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:07.688 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.682+0000 7fa3cbfff640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 0x7fa3b40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.682+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9583+0+0 (secure 0 0 0) 0x7fa3c00cf650 con 0x7fa3cc1011b0 2024-03-28T16:03:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.683+0000 7fa3cbfff640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 0x7fa3b40a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc005fd0 tx=0x7fa3bc009260 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:07.689 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.686+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3c009c020 con 0x7fa3cc1011b0 2024-03-28T16:03:07.933 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.933+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 --> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa3cc108a30 con 0x7fa3b40a2a10 2024-03-28T16:03:07.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.940+0000 7fa3c97fa640 1 -- 172.21.15.23:0/1816240428 <== mgr.15546 v2:172.21.15.23:6800/1692916537 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fa3cc108a30 con 0x7fa3b40a2a10 2024-03-28T16:03:07.940 INFO:teuthology.orchestra.run.smithi023.stdout:/ 1m 6m3h 2024-03-28T16:03:07.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 msgr2=0x7fa3b40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 0x7fa3b40a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc005fd0 tx=0x7fa3bc009260 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 msgr2=0x7fa3cc197070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000ea60 tx=0x7fa3c000ee60 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 shutdown_connections 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fa3b40a2a10 0x7fa3b40a4ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fa3cc105540 0x7fa3cc1975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fa3cc1011b0 0x7fa3cc197070 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 --2- 172.21.15.23:0/1816240428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3cc1007f0 0x7fa3cc196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 >> 172.21.15.23:0/1816240428 conn(0x7fa3cc0773f0 msgr2=0x7fa3cc0fe800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 shutdown_connections 2024-03-28T16:03:07.943 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:07.942+0000 7fa3d27f4640 1 -- 172.21.15.23:0/1816240428 wait complete. 2024-03-28T16:03:07.953 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-28T16:03:07.964 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 -- 172.21.15.23:0/4136843791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 msgr2=0x7f1058107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 --2- 172.21.15.23:0/4136843791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f1058107a30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f1054009930 tx=0x7f105402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:08.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 -- 172.21.15.23:0/4136843791 shutdown_connections 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 --2- 172.21.15.23:0/4136843791 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 0x7f105810e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 --2- 172.21.15.23:0/4136843791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 0x7f1058101ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 --2- 172.21.15.23:0/4136843791 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f1058107a30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 -- 172.21.15.23:0/4136843791 >> 172.21.15.23:0/4136843791 conn(0x7f10580fd130 msgr2=0x7f10580ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 -- 172.21.15.23:0/4136843791 shutdown_connections 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 -- 172.21.15.23:0/4136843791 wait complete. 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.071+0000 7f105fa61640 1 Processor -- start 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 -- start start 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 0x7f1058197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 0x7f10581984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058111a20 con 0x7f1058107650 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10581118a0 con 0x7f1058101640 2024-03-28T16:03:08.074 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058111ba0 con 0x7f1058102000 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39408/0 (socket says 172.21.15.23:39408) 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 -- 172.21.15.23:0/1125786348 learned_addr learned my addr 172.21.15.23:0/1125786348 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105d7fd640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 0x7f1058197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105cffc640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 0x7f10581984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 -- 172.21.15.23:0/1125786348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 msgr2=0x7f10581984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 0x7f10581984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 -- 172.21.15.23:0/1125786348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 msgr2=0x7f1058197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 0x7f1058197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 -- 172.21.15.23:0/1125786348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105819cf50 con 0x7f1058107650 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105dffe640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f105000eb10 tx=0x7f105000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050004b20 con 0x7f1058107650 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1050004cc0 con 0x7f1058107650 2024-03-28T16:03:08.075 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105819d240 con 0x7f1058107650 2024-03-28T16:03:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.072+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1050010760 con 0x7f1058107650 2024-03-28T16:03:08.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.073+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10581a4ad0 con 0x7f1058107650 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.073+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058068490 con 0x7f1058107650 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.075+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1050010900 con 0x7f1058107650 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.075+0000 7f104e7fc640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 0x7f10440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.076+0000 7f105d7fd640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 0x7f10440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.076+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f10500cf9c0 con 0x7f1058107650 2024-03-28T16:03:08.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.076+0000 7f105d7fd640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 0x7f10440a4e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1054005ab0 tx=0x7f1054005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.079+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1050053740 con 0x7f1058107650 2024-03-28T16:03:08.263 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.263+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1058109620 con 0x7f1058107650 2024-03-28T16:03:08.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.264+0000 7f104e7fc640 1 -- 172.21.15.23:0/1125786348 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+25476 (secure 0 0 0) 0x7f105009c300 con 0x7f1058107650 2024-03-28T16:03:08.265 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:08.265 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":101,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T16:02:14.358148+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":13,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1859625110":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/3007922889":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/4155829879":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/421542303":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]},{"pool":4,"snaps":[{"begin":4,"length":1},{"begin":13,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.266+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 msgr2=0x7f10440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.266+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 0x7f10440a4e30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1054005ab0 tx=0x7f1054005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.266+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 msgr2=0x7f105819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.266+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f105000eb10 tx=0x7f105000c2d0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 shutdown_connections 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f10440a2970 0x7f10440a4e30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f1058107650 0x7f105819c820 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1058102000 0x7f10581984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 --2- 172.21.15.23:0/1125786348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1058101640 0x7f1058197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 >> 172.21.15.23:0/1125786348 conn(0x7f10580fd130 msgr2=0x7f10580fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 shutdown_connections 2024-03-28T16:03:08.268 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.267+0000 7f105fa61640 1 -- 172.21.15.23:0/1125786348 wait complete. 2024-03-28T16:03:08.278 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 13, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T16:03:08.279 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:08.391 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 -- 172.21.15.23:0/3161222448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c00688a0 msgr2=0x7f29c0068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 --2- 172.21.15.23:0/3161222448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c00688a0 0x7f29c0068d20 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f29b4009930 tx=0x7f29b402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 -- 172.21.15.23:0/3161222448 shutdown_connections 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 --2- 172.21.15.23:0/3161222448 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c0069260 0x7f29c010e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 --2- 172.21.15.23:0/3161222448 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c00688a0 0x7f29c0068d20 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 --2- 172.21.15.23:0/3161222448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c01076f0 0x7f29c0107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 -- 172.21.15.23:0/3161222448 >> 172.21.15.23:0/3161222448 conn(0x7f29c00776a0 msgr2=0x7f29c0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 -- 172.21.15.23:0/3161222448 shutdown_connections 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.389+0000 7f29c5553640 1 -- 172.21.15.23:0/3161222448 wait complete. 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 Processor -- start 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 -- start start 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 0x7f29c0100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.392 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c0069260 0x7f29c0100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0111a90 con 0x7f29c01076f0 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0111910 con 0x7f29c0069260 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29c5553640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0111c10 con 0x7f29c00688a0 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29beffd640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 0x7f29c0100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:39428/0 (socket says 172.21.15.23:39428) 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 -- 172.21.15.23:0/609773270 learned_addr learned my addr 172.21.15.23:0/609773270 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29be7fc640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c0069260 0x7f29c0100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.393 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 -- 172.21.15.23:0/609773270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 msgr2=0x7f29c0100560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 0x7f29c0100560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 -- 172.21.15.23:0/609773270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c0069260 msgr2=0x7f29c0100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c0069260 0x7f29c0100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29bf7fe640 1 -- 172.21.15.23:0/609773270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c01a6410 con 0x7f29c01076f0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.390+0000 7f29beffd640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 0x7f29c0100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29bf7fe640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f29b800eb10 tx=0x7f29b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8004b20 con 0x7f29c01076f0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29b8004cc0 con 0x7f29c01076f0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c01a6700 con 0x7f29c01076f0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8010760 con 0x7f29c01076f0 2024-03-28T16:03:08.394 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.391+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c01a6c20 con 0x7f29c01076f0 2024-03-28T16:03:08.395 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.392+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c0068490 con 0x7f29c01076f0 2024-03-28T16:03:08.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.393+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f29b8005000 con 0x7f29c01076f0 2024-03-28T16:03:08.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.394+0000 7f29a7fff640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 0x7f29a80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.394+0000 7f29beffd640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 0x7f29a80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.394+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9671+0+0 (secure 0 0 0) 0x7f29b80cfd40 con 0x7f29c01076f0 2024-03-28T16:03:08.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.394+0000 7f29beffd640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 0x7f29a80a4e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009f40 tx=0x7f29b0005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.397 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.397+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f29b809c620 con 0x7f29c01076f0 2024-03-28T16:03:08.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.583+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29c002d070 con 0x7f29c01076f0 2024-03-28T16:03:08.584 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.584+0000 7f29a7fff640 1 -- 172.21.15.23:0/609773270 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29b809bf20 con 0x7f29c01076f0 2024-03-28T16:03:08.584 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 msgr2=0x7f29a80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 0x7f29a80a4e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009f40 tx=0x7f29b0005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 msgr2=0x7f29c0100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f29b800eb10 tx=0x7f29b800c2d0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 shutdown_connections 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f29a80a2970 0x7f29a80a4e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f29c01076f0 0x7f29c0100fe0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c0069260 0x7f29c0100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 --2- 172.21.15.23:0/609773270 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f29c00688a0 0x7f29c0100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 >> 172.21.15.23:0/609773270 conn(0x7f29c00776a0 msgr2=0x7f29c00fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 shutdown_connections 2024-03-28T16:03:08.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.586+0000 7f29c5553640 1 -- 172.21.15.23:0/609773270 wait complete. 2024-03-28T16:03:08.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:08 smithi023 ceph-mon[32448]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:08.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:08 smithi023 ceph-mon[32448]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:08.596 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:08 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1125786348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:08.597 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 -- 172.21.15.23:0/588735153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb01065f0 msgr2=0x7fddb00688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 --2- 172.21.15.23:0/588735153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb01065f0 0x7fddb00688a0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009930 tx=0x7fdda402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 -- 172.21.15.23:0/588735153 shutdown_connections 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 --2- 172.21.15.23:0/588735153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb0104160 0x7fddb01045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 --2- 172.21.15.23:0/588735153 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0068de0 0x7fddb0069240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 --2- 172.21.15.23:0/588735153 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb01065f0 0x7fddb00688a0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 -- 172.21.15.23:0/588735153 >> 172.21.15.23:0/588735153 conn(0x7fddb00776a0 msgr2=0x7fddb0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 -- 172.21.15.23:0/588735153 shutdown_connections 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.710+0000 7fddb6d05640 1 -- 172.21.15.23:0/588735153 wait complete. 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 Processor -- start 2024-03-28T16:03:08.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 -- start start 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0104160 0x7fddb019ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb01065f0 0x7fddb01a2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0118050 con 0x7fddb0068de0 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb0117ed0 con 0x7fddb01065f0 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb6d05640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01181d0 con 0x7fddb0104160 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb4aa1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb4aa1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38934/0 (socket says 172.21.15.23:38934) 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb4aa1640 1 -- 172.21.15.23:0/1510306389 learned_addr learned my addr 172.21.15.23:0/1510306389 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddaffff640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0104160 0x7fddb019ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.711+0000 7fddb52a2640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb01065f0 0x7fddb01a2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 -- 172.21.15.23:0/1510306389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0104160 msgr2=0x7fddb019ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0104160 0x7fddb019ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 -- 172.21.15.23:0/1510306389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb01065f0 msgr2=0x7fddb01a2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb01065f0 0x7fddb01a2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 -- 172.21.15.23:0/1510306389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb01a3500 con 0x7fddb0068de0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb4aa1640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009230 tx=0x7fdda402cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4041070 con 0x7fddb0068de0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdda4004450 con 0x7fddb0068de0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb01a3790 con 0x7fddb0068de0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4003e50 con 0x7fddb0068de0 2024-03-28T16:03:08.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.712+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb01ab080 con 0x7fddb0068de0 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.713+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb0068490 con 0x7fddb0068de0 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.714+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fdda4044070 con 0x7fddb0068de0 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.715+0000 7fddadffb640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 0x7fdda00a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.715+0000 7fddaffff640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 0x7fdda00a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.715+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9671+0+0 (secure 0 0 0) 0x7fdda40f0ea0 con 0x7fddb0068de0 2024-03-28T16:03:08.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.716+0000 7fddaffff640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 0x7fdda00a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fddb0077ed0 tx=0x7fdd9c005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:08.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.719+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdda40bd240 con 0x7fddb0068de0 2024-03-28T16:03:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:08 smithi117 ceph-mon[36002]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:08.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:08 smithi117 ceph-mon[36002]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:08.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:08 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1125786348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:08 smithi120 ceph-mon[36826]: pgmap v310: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 44 KiB/s wr, 29 op/s 2024-03-28T16:03:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:08 smithi120 ceph-mon[36826]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:08.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:08 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1125786348' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:08.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.914+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fddb01ab250 con 0x7fddb0068de0 2024-03-28T16:03:08.915 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.914+0000 7fddadffb640 1 -- 172.21.15.23:0/1510306389 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v102) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdda40c12e0 con 0x7fddb0068de0 2024-03-28T16:03:08.915 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 msgr2=0x7fdda00a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 0x7fdda00a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fddb0077ed0 tx=0x7fdd9c005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 msgr2=0x7fddb019e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009230 tx=0x7fdda402cb30 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 shutdown_connections 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fdda00a2b10 0x7fdda00a4fd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddb01065f0 0x7fddb01a2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fddb0104160 0x7fddb019ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 --2- 172.21.15.23:0/1510306389 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fddb0068de0 0x7fddb019e510 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 >> 172.21.15.23:0/1510306389 conn(0x7fddb00776a0 msgr2=0x7fddb00ff480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:08.917 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 shutdown_connections 2024-03-28T16:03:08.918 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:08.916+0000 7fddb6d05640 1 -- 172.21.15.23:0/1510306389 wait complete. 2024-03-28T16:03:08.928 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 -- 172.21.15.23:0/491032616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0069240 msgr2=0x7fb9c010e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/491032616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0069240 0x7fb9c010e4e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fb9b8009cd0 tx=0x7fb9b802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 -- 172.21.15.23:0/491032616 shutdown_connections 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/491032616 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0069240 0x7fb9c010e4e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/491032616 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0068880 0x7fb9c0068d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/491032616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 0x7fb9c0107a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.034 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 -- 172.21.15.23:0/491032616 >> 172.21.15.23:0/491032616 conn(0x7fb9c00776a0 msgr2=0x7fb9c0077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 -- 172.21.15.23:0/491032616 shutdown_connections 2024-03-28T16:03:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.033+0000 7fb9c52dc640 1 -- 172.21.15.23:0/491032616 wait complete. 2024-03-28T16:03:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 Processor -- start 2024-03-28T16:03:09.035 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 -- start start 2024-03-28T16:03:09.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 0x7fb9c019c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.037 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0111a30 con 0x7fb9c0068880 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c01118b0 con 0x7fb9c01076b0 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9c52dc640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c0111bb0 con 0x7fb9c0069240 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9b67fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9b67fc640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.120:3300/0 says I am v2:172.21.15.23:53106/0 (socket says 172.21.15.23:53106) 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9beffd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38958/0 (socket says 172.21.15.23:38958) 2024-03-28T16:03:09.038 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9b67fc640 1 -- 172.21.15.23:0/2717307073 learned_addr learned my addr 172.21.15.23:0/2717307073 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.034+0000 7fb9bf7fe640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 0x7fb9c019c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 -- 172.21.15.23:0/2717307073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 msgr2=0x7fb9c019c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 0x7fb9c019c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 -- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 msgr2=0x7fb9c0197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.039 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 -- 172.21.15.23:0/2717307073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c019ce60 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9beffd640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9b67fc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800e9f0 tx=0x7fb9a800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a800cd80 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9a8005500 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c019d150 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8010690 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.035+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c0100490 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.036+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c0068490 con 0x7fb9c0069240 2024-03-28T16:03:09.040 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.037+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb9a80046c0 con 0x7fb9c0069240 2024-03-28T16:03:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.038+0000 7fb9bcff9640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 0x7fb9ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.038+0000 7fb9beffd640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 0x7fb9ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.038+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9671+0+0 (secure 0 0 0) 0x7fb9a80cf730 con 0x7fb9c0069240 2024-03-28T16:03:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.038+0000 7fb9beffd640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 0x7fb9ac0a4e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0102750 tx=0x7fb9b0005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.041 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.041+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb9a809c010 con 0x7fb9c0069240 2024-03-28T16:03:09.225 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.225+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb9c0109b80 con 0x7fb9c0069240 2024-03-28T16:03:09.226 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.225+0000 7fb9bcff9640 1 -- 172.21.15.23:0/2717307073 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v102) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb9a809b910 con 0x7fb9c0069240 2024-03-28T16:03:09.226 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T16:03:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 msgr2=0x7fb9ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.228 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 0x7fb9ac0a4e30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb9c0102750 tx=0x7fb9b0005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 msgr2=0x7fb9c01983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800e9f0 tx=0x7fb9a800edf0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 shutdown_connections 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fb9ac0a2970 0x7fb9ac0a4e30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9c01076b0 0x7fb9c019c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fb9c0069240 0x7fb9c01983b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 --2- 172.21.15.23:0/2717307073 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fb9c0068880 0x7fb9c0197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 >> 172.21.15.23:0/2717307073 conn(0x7fb9c00776a0 msgr2=0x7fb9c00ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 shutdown_connections 2024-03-28T16:03:09.229 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.228+0000 7fb9c52dc640 1 -- 172.21.15.23:0/2717307073 wait complete. 2024-03-28T16:03:09.239 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 -- 172.21.15.23:0/519552572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f546010e040 msgr2=0x7f5460110430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 --2- 172.21.15.23:0/519552572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f546010e040 0x7f5460110430 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5458009ca0 tx=0x7f545802c100 comp rx=0 tx=0).stop 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 -- 172.21.15.23:0/519552572 shutdown_connections 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 --2- 172.21.15.23:0/519552572 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f546010e040 0x7f5460110430 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 --2- 172.21.15.23:0/519552572 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f54600691a0 0x7f546010db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.346 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 --2- 172.21.15.23:0/519552572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f5460068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 -- 172.21.15.23:0/519552572 >> 172.21.15.23:0/519552572 conn(0x7f54600776a0 msgr2=0x7f5460077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 -- 172.21.15.23:0/519552572 shutdown_connections 2024-03-28T16:03:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.345+0000 7f546624b640 1 -- 172.21.15.23:0/519552572 wait complete. 2024-03-28T16:03:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 Processor -- start 2024-03-28T16:03:09.347 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 -- start start 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f546019a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546010e040 0x7f546019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5460113c00 con 0x7f54600691a0 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5460113a80 con 0x7f5460068880 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f546624b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5460113d80 con 0x7f546010e040 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f545effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.349 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f545f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f546019a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f545effd640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38976/0 (socket says 172.21.15.23:38976) 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f545effd640 1 -- 172.21.15.23:0/3638620165 learned_addr learned my addr 172.21.15.23:0/3638620165 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.346+0000 7f545ffff640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546010e040 0x7f546019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 -- 172.21.15.23:0/3638620165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546010e040 msgr2=0x7f546019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546010e040 0x7f546019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 -- 172.21.15.23:0/3638620165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 msgr2=0x7f546019a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f546019a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 -- 172.21.15.23:0/3638620165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546019f0a0 con 0x7f54600691a0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545f7fe640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f546019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545effd640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f545400e9f0 tx=0x7f545400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f545400cd80 con 0x7f54600691a0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f546019f390 con 0x7f54600691a0 2024-03-28T16:03:09.350 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5454010910 con 0x7f54600691a0 2024-03-28T16:03:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54540188f0 con 0x7f54600691a0 2024-03-28T16:03:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.347+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601a6c20 con 0x7f54600691a0 2024-03-28T16:03:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.348+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5460068490 con 0x7f54600691a0 2024-03-28T16:03:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.349+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5454010ac0 con 0x7f54600691a0 2024-03-28T16:03:09.351 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.350+0000 7f545cff9640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 0x7f54480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.350+0000 7f545f7fe640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 0x7f54480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.350+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9671+0+0 (secure 0 0 0) 0x7f54540cf550 con 0x7f54600691a0 2024-03-28T16:03:09.352 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.350+0000 7f545f7fe640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 0x7f54480a4e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54500095f0 tx=0x7f5450005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.358 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.358+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f545409b880 con 0x7f54600691a0 2024-03-28T16:03:09.542 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.542+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5460109230 con 0x7f54600691a0 2024-03-28T16:03:09.543 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.542+0000 7f545cff9640 1 -- 172.21.15.23:0/3638620165 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v102) v1 ==== 103+0+11 (secure 0 0 0) 0x7f545409f920 con 0x7f54600691a0 2024-03-28T16:03:09.543 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 msgr2=0x7f54480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 0x7f54480a4e30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f54500095f0 tx=0x7f5450005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 msgr2=0x7f546019a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f545400e9f0 tx=0x7f545400eeb0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 shutdown_connections 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f54480a2970 0x7f54480a4e30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f546010e040 0x7f546019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f54600691a0 0x7f546019a640 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.545 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 --2- 172.21.15.23:0/3638620165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5460068880 0x7f546019a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 >> 172.21.15.23:0/3638620165 conn(0x7f54600776a0 msgr2=0x7f546010f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.544+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 shutdown_connections 2024-03-28T16:03:09.546 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.545+0000 7f546624b640 1 -- 172.21.15.23:0/3638620165 wait complete. 2024-03-28T16:03:09.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:09 smithi023 ceph-mon[32448]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:09.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:09 smithi023 ceph-mon[32448]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T16:03:09.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:09 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/609773270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.556 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:09 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1510306389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.557 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:09 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2717307073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.558 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-28T16:03:09.663 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 -- 172.21.15.23:0/414593483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f664410bce0 msgr2=0x7f664410e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 --2- 172.21.15.23:0/414593483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f664410bce0 0x7f664410e180 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f663c009ca0 tx=0x7f663c02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 -- 172.21.15.23:0/414593483 shutdown_connections 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 --2- 172.21.15.23:0/414593483 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f664410bce0 0x7f664410e180 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 --2- 172.21.15.23:0/414593483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 0x7f664410b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 --2- 172.21.15.23:0/414593483 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f6644102910 0x7f6644102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 -- 172.21.15.23:0/414593483 >> 172.21.15.23:0/414593483 conn(0x7f66440776a0 msgr2=0x7f6644077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 -- 172.21.15.23:0/414593483 shutdown_connections 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 -- 172.21.15.23:0/414593483 wait complete. 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.661+0000 7f664a052640 1 Processor -- start 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 -- start start 2024-03-28T16:03:09.664 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 0x7f6644100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664410bce0 0x7f6644100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644111ae0 con 0x7f6644102910 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644111960 con 0x7f6644103230 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f664a052640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644111c60 con 0x7f664410bce0 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:38992/0 (socket says 172.21.15.23:38992) 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 -- 172.21.15.23:0/1125602315 learned_addr learned my addr 172.21.15.23:0/1125602315 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f6642ffd640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 0x7f6644100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f6643fff640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664410bce0 0x7f6644100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 -- 172.21.15.23:0/1125602315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664410bce0 msgr2=0x7f6644100fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.665 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664410bce0 0x7f6644100fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 -- 172.21.15.23:0/1125602315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 msgr2=0x7f6644100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 0x7f6644100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 -- 172.21.15.23:0/1125602315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66441a63e0 con 0x7f6644102910 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.662+0000 7f66437fe640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f663800ea60 tx=0x7f663800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.663+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663800cde0 con 0x7f6644102910 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.663+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66441a66d0 con 0x7f6644102910 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.663+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66380035f0 con 0x7f6644102910 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.663+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638010670 con 0x7f6644102910 2024-03-28T16:03:09.666 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.663+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66441a6c20 con 0x7f6644102910 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.664+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644068490 con 0x7f6644102910 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.665+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f66380037a0 con 0x7f6644102910 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.665+0000 7f6640ff9640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 0x7f662c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.665+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9671+0+0 (secure 0 0 0) 0x7f66380cf650 con 0x7f6644102910 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.665+0000 7f6642ffd640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 0x7f662c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:09.667 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.666+0000 7f6642ffd640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 0x7f662c0a4e60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6644101fa0 tx=0x7f6634005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:09.670 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.670+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f663809bfc0 con 0x7f6644102910 2024-03-28T16:03:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:09 smithi117 ceph-mon[36002]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:09 smithi117 ceph-mon[36002]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T16:03:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:09 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/609773270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:09 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1510306389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:09 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2717307073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:09 smithi120 ceph-mon[36826]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-28T16:03:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:09 smithi120 ceph-mon[36826]: osdmap e102: 12 total, 12 up, 12 in 2024-03-28T16:03:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:09 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/609773270' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:09 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1510306389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:09 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2717307073' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:09.904 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.904+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f6644101a40 con 0x7f6644102910 2024-03-28T16:03:09.912 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.912+0000 7f6640ff9640 1 -- 172.21.15.23:0/1125602315 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v25) v1 ==== 81+0+740361 (secure 0 0 0) 0x7f663809b8c0 con 0x7f6644102910 2024-03-28T16:03:09.922 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:09.922 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T16:03:09.922 INFO:teuthology.orchestra.run.smithi023.stdout: "epoch": 25, 2024-03-28T16:03:09.922 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.922 INFO:teuthology.orchestra.run.smithi023.stdout: "active_gid": 15546, 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "active_name": "x", 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addrs": { 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6800", 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1692916537 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v1", 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:6801", 2024-03-28T16:03:09.923 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 1692916537 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "active_addr": "172.21.15.23:6801/1692916537", 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "active_change": "2024-03-28T15:53:05.125616+0000", 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "active_mgr_features": 4540701547738038271, 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "available": true, 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "standbys": [ 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 15543, 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "y", 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T16:03:09.924 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.925 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.926 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.930 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.930 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.930 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.930 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:09.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.932 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:09.933 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.935 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T16:03:09.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T16:03:09.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.943 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.944 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:09.946 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T16:03:09.947 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:03:09.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T16:03:09.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:09.952 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:09.953 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:09.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.956 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T16:03:09.959 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:09.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.962 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.964 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:09.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.966 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.967 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.969 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.970 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:03:09.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.973 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:03:09.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.976 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.978 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T16:03:09.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.990 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:03:09.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:09.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:09.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.996 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:09.998 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:09.999 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.000 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.002 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.006 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.007 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.010 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.011 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:03:10.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:03:10.014 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T16:03:10.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T16:03:10.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T16:03:10.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:10.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.019 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.020 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T16:03:10.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T16:03:10.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T16:03:10.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.025 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.029 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:03:10.032 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:03:10.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.034 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.035 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.038 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.039 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.044 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.047 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.048 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.052 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:03:10.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.056 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:03:10.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:03:10.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.060 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T16:03:10.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:03:10.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:03:10.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:03:10.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:03:10.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:03:10.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:03:10.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:03:10.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:03:10.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.070 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.071 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:03:10.075 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:03:10.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.078 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:03:10.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.082 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.085 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.091 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.093 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.096 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T16:03:10.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:10.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.100 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.101 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.103 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.113 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.115 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.117 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.119 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.120 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T16:03:10.122 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:03:10.123 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.124 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T16:03:10.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T16:03:10.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.127 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.128 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.129 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.130 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.131 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.132 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.133 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.134 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.135 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.139 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.142 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T16:03:10.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.145 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.150 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.158 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.159 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.162 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.164 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.165 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.168 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.170 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.171 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T16:03:10.172 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:03:10.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.175 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.178 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.180 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.182 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T16:03:10.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.189 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:03:10.190 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.191 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T16:03:10.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T16:03:10.193 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.197 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.198 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.199 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.200 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.201 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.202 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.204 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.207 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.209 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.211 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.213 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.217 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T16:03:10.218 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.219 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.220 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.223 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T16:03:10.225 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.226 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.227 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.230 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.232 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.233 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.235 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.236 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.240 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.241 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.245 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:03:10.250 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T16:03:10.252 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T16:03:10.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.259 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.261 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.262 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:10.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.270 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.271 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.274 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:03:10.276 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.279 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.280 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.283 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.285 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.286 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.287 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.289 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.292 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.293 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.295 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.296 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.298 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.299 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.302 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.304 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.306 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.307 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T16:03:10.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T16:03:10.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.318 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.319 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T16:03:10.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T16:03:10.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.326 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.329 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.330 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.332 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.333 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.335 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.339 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.341 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.342 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.344 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.350 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.354 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.355 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.356 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.357 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.358 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.359 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.360 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.361 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T16:03:10.362 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:10.363 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.364 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T16:03:10.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.366 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.367 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.369 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.370 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.371 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.372 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.373 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.374 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.375 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T16:03:10.376 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.377 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.378 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.379 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.380 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.381 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.382 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.383 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.384 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.385 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.386 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.387 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.388 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.389 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.390 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.391 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.392 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.393 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:03:10.394 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T16:03:10.395 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T16:03:10.396 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.397 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.398 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.399 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.400 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.401 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.402 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.403 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.404 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.405 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.406 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: "gid": 25327, 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "z", 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: "mgr_features": 4540701547738038271, 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.407 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:03:10.408 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.409 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.410 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.411 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.412 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.413 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.414 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.415 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.416 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T16:03:10.417 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.418 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T16:03:10.419 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.420 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.421 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.422 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.423 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.424 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.425 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.426 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.427 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:10.428 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.429 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:03:10.430 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:03:10.431 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:10.432 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:10.433 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.434 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.435 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.436 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.437 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.438 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.439 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.440 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.441 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.442 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.443 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.444 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.445 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.446 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.447 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.448 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.449 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.450 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.451 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.452 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T16:03:10.453 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.454 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.455 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.456 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.457 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.458 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.459 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.460 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T16:03:10.461 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.462 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.463 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.464 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:03:10.465 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:03:10.466 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T16:03:10.467 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T16:03:10.468 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T16:03:10.469 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.470 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.471 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T16:03:10.472 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T16:03:10.473 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.474 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.475 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:03:10.476 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.477 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.478 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.479 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.480 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.481 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.482 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.483 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.484 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.485 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.486 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.487 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:03:10.488 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.489 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.491 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.492 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.493 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T16:03:10.494 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.495 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:03:10.496 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.497 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:10.498 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T16:03:10.499 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.500 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.501 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.502 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.503 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.504 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.505 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.506 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.507 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:03:10.508 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:03:10.509 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:03:10.510 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:03:10.511 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:03:10.512 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:03:10.513 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.514 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.515 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.516 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.517 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.518 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.519 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.520 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.521 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:03:10.522 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.523 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.524 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.525 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.526 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.527 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.528 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.529 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.530 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.531 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.532 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.533 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.534 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.535 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.536 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.537 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.538 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:03:10.539 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:03:10.539 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.542 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.543 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.544 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.545 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.546 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.547 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.548 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.549 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.550 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.551 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:03:10.552 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:03:10.553 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:03:10.554 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:03:10.555 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.556 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.557 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.558 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.559 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.560 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.561 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.562 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.563 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.564 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.565 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.566 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.567 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.568 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.569 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:03:10.570 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:03:10.571 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.572 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.573 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.574 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.575 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.576 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.577 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.578 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.579 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.580 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.581 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.582 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.583 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.584 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.585 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.586 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.587 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.588 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.589 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.590 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.591 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.592 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.593 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.594 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.595 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T16:03:10.596 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.597 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.598 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.599 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.600 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.601 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.602 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.603 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.604 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:03:10.605 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.606 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.607 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T16:03:10.608 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.609 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.610 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.611 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.612 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.613 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.614 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.615 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.616 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.617 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:10.618 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.619 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.620 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.621 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.622 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.623 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T16:03:10.624 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.625 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.626 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.627 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.628 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.629 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.630 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.631 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.632 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.633 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T16:03:10.634 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.635 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T16:03:10.636 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.637 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.638 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.639 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.640 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.641 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.642 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.643 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.644 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.645 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.646 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.647 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.648 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.649 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.650 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:03:10.651 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.652 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.653 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.654 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.656 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.657 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.658 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.659 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.660 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.661 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.662 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.663 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.664 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.665 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.666 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.667 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.668 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.669 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.670 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:03:10.671 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:03:10.672 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.673 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.674 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.675 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.676 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.677 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.678 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.679 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.680 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.681 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.682 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.683 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T16:03:10.684 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.685 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.686 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.687 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.688 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.689 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.690 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.691 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.692 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.693 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.694 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.695 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.696 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.697 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.698 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.699 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T16:03:10.700 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.701 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.702 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.703 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.704 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.705 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.706 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.707 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.708 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.709 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.710 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.711 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.712 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.713 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.714 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.715 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.716 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.717 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.718 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.719 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T16:03:10.720 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.721 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.722 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.723 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.724 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.725 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.726 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.727 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.728 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:03:10.729 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.730 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.731 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.732 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T16:03:10.733 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:03:10.734 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.735 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.736 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.737 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.738 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.739 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.740 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T16:03:10.741 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.742 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.743 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.746 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.746 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.746 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.747 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.748 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.749 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T16:03:10.750 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.751 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T16:03:10.752 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.753 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.754 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.755 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.756 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.757 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.758 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.759 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.760 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.761 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.762 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.763 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.764 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.765 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.766 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.767 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.768 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.769 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.770 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.771 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.772 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.773 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.774 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.775 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.776 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.777 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.778 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.779 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.780 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.781 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.782 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.783 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.784 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.785 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.786 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.787 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.788 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.789 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.790 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T16:03:10.791 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T16:03:10.792 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T16:03:10.793 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T16:03:10.794 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.795 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T16:03:10.796 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T16:03:10.797 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T16:03:10.798 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T16:03:10.799 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T16:03:10.800 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T16:03:10.801 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T16:03:10.802 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.803 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.804 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.805 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.806 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.807 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.808 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.809 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.810 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.811 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.812 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.813 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.814 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.815 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.816 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.817 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.818 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.819 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.820 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.821 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.822 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.823 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.824 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.825 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.826 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.827 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.828 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.829 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.830 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.831 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.832 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.833 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.834 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.835 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.836 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:03:10.837 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:03:10.838 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.839 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.840 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.841 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.842 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.843 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.844 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T16:03:10.845 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.846 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.847 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.848 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.849 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.850 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.851 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.852 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.853 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.854 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.855 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.856 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.857 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.858 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.859 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.860 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.861 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.862 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.863 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.864 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.865 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.866 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.867 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.868 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.869 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.870 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.871 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.872 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:03:10.873 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.874 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.875 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.876 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.877 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.878 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.879 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.880 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.881 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.882 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.883 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.884 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.885 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.886 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "modules": [ 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard", 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "iostat", 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "nfs", 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "restful", 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: "snap_schedule" 2024-03-28T16:03:10.887 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "available_modules": [ 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "alerts", 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.888 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How frequently to reexamine health status", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.889 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.890 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.891 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.892 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.893 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.894 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.895 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_destination": { 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_destination", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.896 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email address to send alerts to", 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_from_name": { 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_from_name", 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.897 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "Ceph", 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Email From: name", 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_host": { 2024-03-28T16:03:10.898 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_host", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP server", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_password": { 2024-03-28T16:03:10.899 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_password", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Password to authenticate with", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_port": { 2024-03-28T16:03:10.900 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_port", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "465", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP port", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_sender": { 2024-03-28T16:03:10.901 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_sender", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SMTP envelope sender", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.902 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_ssl": { 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_ssl", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.903 INFO:teuthology.orchestra.run.smithi023.stdout: "smtp_user": { 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "smtp_user", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "User to authenticate as", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.904 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "balancer", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "active": { 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "active", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.905 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically balance PGs across cluster", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_time": { 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_time", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0000", 2024-03-28T16:03:10.906 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "beginning time of day to automatically balance", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "begin_weekday": { 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "begin_weekday", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.907 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_max_iterations": { 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_max_iterations", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.908 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "25", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "250", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_metrics": { 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_metrics", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.909 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "pgs,objects,bytes", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "crush_compat_step": { 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crush_compat_step", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.910 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.5", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0.001", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "0.999", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "aggressiveness of optimization", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "end_time": { 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_time", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.911 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2359", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ending time of day to automatically balance", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "end_weekday": { 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "end_weekday", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.912 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "0", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "6", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.913 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.914 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.915 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.916 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.917 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.918 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min_score": { 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_score", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.919 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "upmap", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "crush-compat", 2024-03-28T16:03:10.920 INFO:teuthology.orchestra.run.smithi023.stdout: "none", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "read", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap-read" 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Balancer mode", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_ids": { 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_ids", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.921 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.922 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.923 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_deviation": { 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_deviation", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.924 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "upmap_max_optimizations": { 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "upmap_max_optimizations", 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.925 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm", 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.926 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_down_multiplier": { 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_down_multiplier", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-28T16:03:10.927 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_refresh_rate": { 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_refresh_rate", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "20", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.928 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "agent_starting_port": { 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "agent_starting_port", 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.929 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4721", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_ptrace": { 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_ptrace", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.930 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_interval": { 2024-03-28T16:03:10.931 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_interval", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.932 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "autotune_memory_target_ratio": { 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "autotune_memory_target_ratio", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0.7", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.933 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm_log_destination": { 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cephadm_log_destination", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.934 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "file", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "file,syslog", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "syslog" 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.935 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "cgroups_split": { 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cgroups_split", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.936 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "config_checks_enabled": { 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_checks_enabled", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.937 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "config_dashboard": { 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_dashboard", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.938 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_alertmanager": { 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_alertmanager", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.939 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_base": { 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_base", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.940 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Container image name, without the tag", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_elasticsearch": { 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_elasticsearch", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.941 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "elasticsearch container image", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_grafana": { 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_grafana", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.942 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_haproxy": { 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_haproxy", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.943 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "HAproxy container image", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_agent": { 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_agent", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.944 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger agent container image", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_collector": { 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_collector", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.945 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger collector container image", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_jaeger_query": { 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_jaeger_query", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.946 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Jaeger query container image", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_keepalived": { 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_keepalived", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.947 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Keepalived container image", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_loki": { 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_loki", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.948 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Loki container image", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_node_exporter": { 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_node_exporter", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.949 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_nvmeof": { 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_nvmeof", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.950 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Nvme-of container image", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_prometheus": { 2024-03-28T16:03:10.951 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_prometheus", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Prometheus container image", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.952 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_promtail": { 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_promtail", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Promtail container image", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.953 INFO:teuthology.orchestra.run.smithi023.stdout: "container_image_snmp_gateway": { 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_image_snmp_gateway", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "SNMP Gateway container image", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.954 INFO:teuthology.orchestra.run.smithi023.stdout: "container_init": { 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "container_init", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Run podman/docker with `--init`", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.955 INFO:teuthology.orchestra.run.smithi023.stdout: "daemon_cache_timeout": { 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "daemon_cache_timeout", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.956 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_cephadm_command_timeout": { 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_cephadm_command_timeout", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "900", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.957 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_registry": { 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "default_registry", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "docker.io", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.958 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "device_cache_timeout": { 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_cache_timeout", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1800", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache device inventory", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.959 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "device_enhanced_scan": { 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_enhanced_scan", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.960 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "facts_cache_timeout": { 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "facts_cache_timeout", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.961 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "seconds to cache host facts data", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "host_check_interval": { 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "host_check_interval", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.962 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to perform a host check", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "hw_monitoring": { 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hw_monitoring", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.963 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "inventory_list_all": { 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "inventory_list_all", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.964 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.965 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.966 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "log_refresh_metadata": { 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_refresh_metadata", 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.967 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.968 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.969 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.970 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.971 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.972 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-28T16:03:10.973 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "*", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.974 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max_count_per_host": { 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_count_per_host", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.975 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of daemons per service per host", 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "max_osd_draining_count": { 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_osd_draining_count", 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.976 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.977 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "migration_current": { 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "migration_current", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "internal - do not modify", 2024-03-28T16:03:10.978 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "mode": { 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mode", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "root", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm-package", 2024-03-28T16:03:10.979 INFO:teuthology.orchestra.run.smithi023.stdout: "root" 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "mode for remote execution of cephadm", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "oob_default_addr": { 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "oob_default_addr", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "169.254.1.1", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.980 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "prometheus_alerts_path": { 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus_alerts_path", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.981 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_insecure": { 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_insecure", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.982 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_password": { 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_password", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.983 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_url": { 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_url", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.984 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "registry_username": { 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "registry_username", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.985 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "secure_monitoring_stack": { 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "secure_monitoring_stack", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.986 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "service_discovery_port": { 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "service_discovery_port", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8765", 2024-03-28T16:03:10.987 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "cephadm service discovery port", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "ssh_config_file": { 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssh_config_file", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.988 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "use_agent": { 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_agent", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.989 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "use_repo_digest": { 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "use_repo_digest", 2024-03-28T16:03:10.990 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_failed_host_check": { 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_failed_host_check", 2024-03-28T16:03:10.991 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if the host check fails", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_daemons": { 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_daemons", 2024-03-28T16:03:10.992 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_on_stray_hosts": { 2024-03-28T16:03:10.993 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_on_stray_hosts", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.994 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crash", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.995 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.996 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.997 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:10.998 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:10.999 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "retain_interval": { 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retain_interval", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "31536000", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.000 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_recent_interval": { 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_recent_interval", 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.001 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1209600", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.002 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dashboard", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:11.003 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_HOST": { 2024-03-28T16:03:11.004 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-28T16:03:11.005 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.006 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_ENABLED": { 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_ENABLED", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.007 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.008 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "ENABLE_BROWSABLE_API": { 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.009 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.010 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.011 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.012 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.013 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.014 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.014 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.014 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.014 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.014 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.015 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.016 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.017 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.018 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-28T16:03:11.019 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_PASSWORD": { 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-28T16:03:11.020 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-28T16:03:11.021 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_URL": { 2024-03-28T16:03:11.022 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_URL", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_API_USERNAME": { 2024-03-28T16:03:11.023 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.024 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.025 INFO:teuthology.orchestra.run.smithi023.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.026 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.027 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.028 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "MULTICLUSTER_CONFIG": { 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "{}", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.029 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_HOST": { 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.030 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.031 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.032 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.033 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.034 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.035 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.036 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.037 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.038 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_ENABLED": { 2024-03-28T16:03:11.039 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.040 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.041 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.042 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.043 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "45", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.044 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ACCESS_KEY": { 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.045 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.046 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "admin", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SECRET_KEY": { 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.047 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.048 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "RGW_API_SSL_VERIFY": { 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.049 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "UNSAFE_TLS_v1_2": { 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.050 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.051 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.052 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.053 INFO:teuthology.orchestra.run.smithi023.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.054 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "cross_origin_url": { 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cross_origin_url", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.055 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.056 INFO:teuthology.orchestra.run.smithi023.stdout: "crt_file": { 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "crt_file", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.057 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "debug": { 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "debug", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Enable/disable debug options", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.058 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "jwt_token_ttl": { 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "jwt_token_ttl", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "28800", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.059 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.060 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.061 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.062 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.063 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.064 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.065 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "motd": { 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "motd", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.066 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "The message of the day", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect_resolve_ip_addr": { 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "redirect_resolve_ip_addr", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.067 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:11.068 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8080", 2024-03-28T16:03:11.069 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.070 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl_server_port": { 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl_server_port", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.071 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8443", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.072 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "redirect", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "redirect" 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:11.073 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "url_prefix": { 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url_prefix", 2024-03-28T16:03:11.074 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.075 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_monitoring": { 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_monitoring", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.076 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "monitor device health metrics", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.077 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.078 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.079 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.080 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.081 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.082 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "mark_out_threshold": { 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mark_out_threshold", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "2419200", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.083 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "pool_name": { 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pool_name", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "device_health_metrics", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.084 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "retention_period": { 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "retention_period", 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.085 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15552000", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long to retain device health metrics", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_frequency": { 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_frequency", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.086 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.087 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "self_heal": { 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "self_heal", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.088 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.089 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how frequently to wake up and check device health", 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "warn_threshold": { 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "warn_threshold", 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.090 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7257600", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "diskprediction_local", 2024-03-28T16:03:11.091 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.092 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.093 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.094 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.095 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.096 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "predict_interval": { 2024-03-28T16:03:11.097 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predict_interval", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "86400", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.098 INFO:teuthology.orchestra.run.smithi023.stdout: "predictor_model": { 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "predictor_model", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "prophetstor", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.099 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "600", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.100 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "influx", 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": false, 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "influxdb python module not found", 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "batch_size": { 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "batch_size", 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.101 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5000", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "database": { 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "database", 2024-03-28T16:03:11.102 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "ceph", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "hostname": { 2024-03-28T16:03:11.103 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "hostname", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server hostname", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.104 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "30", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "5", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.105 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.106 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.107 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.108 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.109 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.110 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "password": { 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "password", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "password of InfluxDB server user", 2024-03-28T16:03:11.111 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "port": { 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "port", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "8086", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "InfluxDB server port", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.112 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "ssl": { 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ssl", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "false", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:11.113 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "threads": { 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threads", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "32", 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.114 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "username": { 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "username", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.115 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "username of InfluxDB server user", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "verify_ssl": { 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "verify_ssl", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "true", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.116 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.117 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "insights", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.118 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.119 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.120 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.121 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.122 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.123 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "iostat", 2024-03-28T16:03:11.124 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.125 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.126 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.127 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.128 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.129 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.130 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "k8sevents", 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_event_retention_days": { 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "ceph_event_retention_days", 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.131 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "7", 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "config_check_secs": { 2024-03-28T16:03:11.132 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "config_check_secs", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "10", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.133 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.134 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.135 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.136 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.137 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 msgr2=0x7f662c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.137 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.138 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.139 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "localpool", 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "failure_domain": { 2024-03-28T16:03:11.140 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "failure_domain", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "host", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "failure domain for any created local pool", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.141 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.142 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.143 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.144 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.145 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.146 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min_size": { 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "min_size", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default min_size for any created local pool", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.147 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "num_rep": { 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "num_rep", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default replica count for any created local pool", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.148 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_num": { 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_num", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "128", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "default pg_num for any created local pool", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.149 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "prefix": { 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prefix", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "name prefix for any created local pool", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.150 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "subtree": { 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "subtree", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "rack", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-28T16:03:11.151 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mds_autoscaler", 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.152 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.153 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.154 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.155 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.156 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.157 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirroring", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.158 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.159 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.160 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.161 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.162 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.163 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "nfs", 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.164 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.165 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.166 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.167 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.168 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.169 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.170 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.171 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "fail_fs": { 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "fail_fs", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.172 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.173 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.174 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.175 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.176 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.177 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator": { 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "orchestrator", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.178 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "cephadm", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "rook", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "test_orchestrator" 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Orchestrator backend", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_perf_query", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.180 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.181 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.182 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.183 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.184 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "osd_support", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.185 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.186 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.187 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.188 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.189 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.190 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "pg_autoscaler", 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.191 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.192 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.193 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.194 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.195 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:11.196 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "threshold": { 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "threshold", 2024-03-28T16:03:11.197 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "3.0", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1.0", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "scaling threshold", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.198 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "progress", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_pg_recovery_event": { 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_pg_recovery_event", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.199 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.200 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.201 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.202 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.203 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.204 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.205 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "max_completed_events": { 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_completed_events", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "50", 2024-03-28T16:03:11.206 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "number of past completed events to remember", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "sleep_interval": { 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "sleep_interval", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.207 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "5", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "how long the module is going to sleep", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "prometheus", 2024-03-28T16:03:11.208 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "cache": { 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "cache", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.209 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "exclude_perf_counters": { 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "exclude_perf_counters", 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.210 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 0x7f662c0a4e60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6644101fa0 tx=0x7f6634005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 msgr2=0x7f6644100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f663800ea60 tx=0x7f663800ef20 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 shutdown_connections 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f662c0a29a0 0x7f662c0a4e60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f664410bce0 0x7f6644100fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6644103230 0x7f6644100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 --2- 172.21.15.23:0/1125602315 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f6644102910 0x7f6644100560 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:11.211 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 >> 172.21.15.23:0/1125602315 conn(0x7f66440776a0 msgr2=0x7f66440fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.212 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.213 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.214 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.215 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.216 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.217 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools": { 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.218 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "300", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.219 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "scrape_interval": { 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "scrape_interval", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15.0", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.220 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "::", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.221 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "9283", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.222 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "stale_cache_strategy": { 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stale_cache_strategy", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "log", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.223 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_behaviour": { 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_behaviour", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "default", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.224 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "default", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "error" 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "standby_error_status_code": { 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "standby_error_status_code", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.225 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "500", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "400", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "599", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T16:03:11.226 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.227 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.228 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.229 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.230 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.231 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_snap_create": { 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_snap_create", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.232 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "mirror_snapshot_schedule": { 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "mirror_snapshot_schedule", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.233 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "trash_purge_schedule": { 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "trash_purge_schedule", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.234 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "restful", 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.235 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enable_auth": { 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enable_auth", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.236 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "key_file": { 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "key_file", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.237 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.238 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.239 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.240 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.241 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.242 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "server_addr": { 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_addr", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.243 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "server_port": { 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "server_port", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.244 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rgw", 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.245 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.246 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.247 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.248 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.249 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.250 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rook", 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.251 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.252 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.253 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.254 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.255 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.256 INFO:teuthology.orchestra.run.smithi023.stdout: "storage_class": { 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "storage_class", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "local", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.257 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "selftest", 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.258 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.259 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.260 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.261 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.262 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.263 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "roption1": { 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption1", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.264 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "roption2": { 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "roption2", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "xyz", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.265 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption1": { 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption1", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.266 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption2": { 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption2", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.267 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption3": { 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption3", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "float", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.268 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption4": { 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption4", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.269 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption5": { 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption5", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.270 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption6": { 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption6", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.271 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "rwoption7": { 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rwoption7", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.272 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "1", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "42", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "testkey": { 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testkey", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.273 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "testlkey": { 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testlkey", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.274 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "testnewline": { 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "testnewline", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.275 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.276 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "allow_m_granularity": { 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "allow_m_granularity", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "allow minute scheduled snapshots", 2024-03-28T16:03:11.277 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "dump_on_update": { 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "dump_on_update", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "dump database to debug log on update", 2024-03-28T16:03:11.278 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.279 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.280 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.281 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.282 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.283 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.283 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.283 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.283 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 shutdown_connections 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.284 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "stats", 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.285 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.286 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.287 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.288 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.289 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.290 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "status", 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.291 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.292 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.293 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.294 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.295 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telegraf", 2024-03-28T16:03:11.296 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "address": { 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "address", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.297 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "15", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.298 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.299 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.300 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.301 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.302 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.303 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "telemetry", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_basic": { 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_basic", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.304 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_crash": { 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_crash", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.305 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_device": { 2024-03-28T16:03:11.306 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_device", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_ident": { 2024-03-28T16:03:11.307 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_ident", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.308 INFO:teuthology.orchestra.run.smithi023.stdout: "channel_perf": { 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "channel_perf", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "contact": { 2024-03-28T16:03:11.309 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "contact", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "description": { 2024-03-28T16:03:11.310 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "description", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "device_url": { 2024-03-28T16:03:11.311 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "device_url", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "enabled": { 2024-03-28T16:03:11.312 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "enabled", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:11.313 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "24", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "8", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "last_opt_revision": { 2024-03-28T16:03:11.314 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "last_opt_revision", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "1", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard": { 2024-03-28T16:03:11.315 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "leaderboard_description": { 2024-03-28T16:03:11.316 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "leaderboard_description", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.317 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.318 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.319 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.320 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.321 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.322 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "organization": { 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "organization", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.323 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "proxy": { 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "proxy", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.324 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "url": { 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "url", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.325 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "test_orchestrator", 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.326 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.327 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.328 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.329 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.330 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.331 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.332 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.333 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.334 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.335 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.336 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "max_concurrent_clones": { 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "max_concurrent_clones", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.337 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "4", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "periodic_async_work": { 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "periodic_async_work", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.338 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Periodically check for async work", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_delay": { 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_delay", 2024-03-28T16:03:11.339 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "0", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "snapshot_clone_no_wait": { 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snapshot_clone_no_wait", 2024-03-28T16:03:11.340 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "True", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.341 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "can_run": true, 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "error_string": "", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "module_options": { 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "discovery_interval": { 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "discovery_interval", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "uint", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "100", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.342 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "identifier": { 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "identifier", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.343 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "interval": { 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "interval", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "secs", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "60", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.344 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "log_level": { 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_level", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.345 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.346 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster": { 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.347 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_cluster_level": { 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_cluster_level", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "info", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [ 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "critical", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "debug", 2024-03-28T16:03:11.348 INFO:teuthology.orchestra.run.smithi023.stdout: "error", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "info", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "warning" 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "log_to_file": { 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "log_to_file", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "bool", 2024-03-28T16:03:11.349 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 1, 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "False", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_host": { 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_host", 2024-03-28T16:03:11.350 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_port": { 2024-03-28T16:03:11.351 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_port", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "int", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "10051", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "zabbix_sender": { 2024-03-28T16:03:11.352 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "zabbix_sender", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "str", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "level": "advanced", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "flags": 0, 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "min": "", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "max": "", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "enum_allowed": [], 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "desc": "", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "long_desc": "", 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "tags": [], 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: "see_also": [] 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.353 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.354 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.354 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.354 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:09.917+0000 7f664a052640 1 -- 172.21.15.23:0/1125602315 wait complete. 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:10 smithi120 ceph-mon[36826]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:10 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3638620165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:10 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1125602315' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:10 smithi117 ceph-mon[36002]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:10 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3638620165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:11.359 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:10 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1125602315' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "services": { 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "dashboard": "https://172.21.15.23:8443/" 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "always_on_modules": { 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "octopus": [ 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T16:03:11.360 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "pacific": [ 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T16:03:11.361 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "quincy": [ 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T16:03:11.362 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "reef": [ 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T16:03:11.363 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: ], 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "squid": [ 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "balancer", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "crash", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "devicehealth", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "orchestrator", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "pg_autoscaler", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "progress", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "rbd_support", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "status", 2024-03-28T16:03:11.364 INFO:teuthology.orchestra.run.smithi023.stdout: "telemetry", 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "volumes" 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "last_failure_osd_epoch": 95, 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "active_clients": [ 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "devicehealth", 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 288517520 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.365 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "libcephsqlite", 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3294784990 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.366 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "rbd_support", 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3492418446 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.367 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 3428449458 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "snap_schedule", 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 2737861929 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.368 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: "name": "volumes", 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addrvec": [ 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: { 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: "type": "v2", 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: "addr": "172.21.15.23:0", 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: "nonce": 872468011 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout: ] 2024-03-28T16:03:11.369 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T16:03:11.375 DEBUG:tasks.check_counter:Getting stats from a 2024-03-28T16:03:11.376 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-28T16:03:11.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:10 smithi023 ceph-mon[32448]: pgmap v312: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 53 KiB/s wr, 30 op/s 2024-03-28T16:03:11.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:10 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3638620165' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:11.376 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:10 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1125602315' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-28T16:03:11.624 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:12.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:12 smithi023 ceph-mon[32448]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:12.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:12 smithi117 ceph-mon[36002]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:12.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:12 smithi120 ceph-mon[36826]: pgmap v313: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:13.465 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T16:03:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:13.465 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 722, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 942, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1624258, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 685640, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 10, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 4, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.080230778, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.031127867, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.053811163, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.006319726, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 942, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.046481270, 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000049343 2024-03-28T16:03:13.466 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 963, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000165687, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000172 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1488160, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 436288 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 80, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 146, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 26767, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 215741, 2024-03-28T16:03:13.467 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 16, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.020658318, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009459958, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.009613295, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000178543, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 143, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.030632226, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000214211 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 158, 2024-03-28T16:03:13.468 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000027870, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000176 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1088, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 131936 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 78, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 82, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 27990, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 33701, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 9, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.007348478, 2024-03-28T16:03:13.469 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.001941522, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.005078207, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000472269, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 80, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.021762752, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000272034 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 101, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000024449, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000242 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-03-28T16:03:13.470 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 2864 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 62, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.001626238, 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000026229 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.471 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000011727, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000005863 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 6, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 4, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 4, 2024-03-28T16:03:13.472 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007293164, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001823291 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 2, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 11, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 14, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 14, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:13.473 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 14, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 2, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 2, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 2, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 8, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 5, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 1, 2024-03-28T16:03:13.474 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 1, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 4, 2024-03-28T16:03:13.475 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 234, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 3, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 2, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 3, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:13.476 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 2, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 1 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 2, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:13.477 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-28T16:03:13.478 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 7, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 7, 2024-03-28T16:03:13.479 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 2, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 2, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 2, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 4196264, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 0, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 6, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.010648918, 2024-03-28T16:03:13.480 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.001774819 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 0 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 17, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 17, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 0, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 15, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 16, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 1, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 14, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 14, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-28T16:03:13.481 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 2, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 4, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 2, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 52540, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 223612 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 7, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 6, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 17, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.482 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.006894092, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.003447046 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.483 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000204200, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000204200 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.484 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000194872, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000194872 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.485 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.486 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.487 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-28T16:03:13.488 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.489 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.490 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.491 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.492 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.493 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.494 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 1, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 3, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 2, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 1, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 3, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 1, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 615, 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:13.495 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:13.496 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:13.497 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 295713, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 105, 2024-03-28T16:03:13.498 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 10040, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 194, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T16:03:13.499 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 67349, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 294, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 46, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 12851, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 33, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 46, 2024-03-28T16:03:13.500 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 46, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.236997978, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.005152129 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 46, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 72 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 46, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 0, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 46, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-28T16:03:13.501 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 10, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 6, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 7, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 6, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:13.502 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 20, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-28T16:03:13.503 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-28T16:03:13.504 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 102, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 12, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 9, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 9, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 22, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 0, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 1 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.505 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 1, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-28T16:03:13.506 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 876, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1583114, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 876, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:13.507 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 876, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1583114, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:13.508 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 46, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 12851, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 46, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 12851, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:13.509 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 46, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 46, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 46, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 46, 2024-03-28T16:03:13.510 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:13.511 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x56388e9ac500": { 2024-03-28T16:03:13.512 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 7, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1960, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 7, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 6, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1960, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:13.513 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.514 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:13.514 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T16:03:14.201 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-28T16:03:14.201 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-28T16:03:14.201 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=3 2024-03-28T16:03:14.201 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:14.201 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:14.201 DEBUG:tasks.check_counter:Getting stats from d 2024-03-28T16:03:14.201 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-28T16:03:14.465 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:14.533 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:14 smithi023 ceph-mon[32448]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:14.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:14 smithi117 ceph-mon[36002]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:14.843 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:14 smithi120 ceph-mon[36826]: pgmap v314: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:15.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:15 smithi023 ceph-mon[32448]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T16:03:15.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:15 smithi117 ceph-mon[36002]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T16:03:15.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:15 smithi120 ceph-mon[36826]: osdmap e103: 12 total, 12 up, 12 in 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 4, 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 4, 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 2992, 2024-03-28T16:03:16.264 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 448, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001604594, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000029977, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000562350, 2024-03-28T16:03:16.265 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T16:03:16.266 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:16.266 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3, 2024-03-28T16:03:16.266 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000689207, 2024-03-28T16:03:16.266 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000229735 2024-03-28T16:03:16.266 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.273 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 223, 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 213, 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1437904, 2024-03-28T16:03:16.274 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 127440, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.038183549, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009838910, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.034301084, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003617237, 2024-03-28T16:03:16.275 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 212, 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.006529803, 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000030800 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:16.276 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1437904, 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 127440 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.277 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 163, 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183808, 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T16:03:16.278 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.010887680, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.007135877, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001877392, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 160, 2024-03-28T16:03:16.279 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.008547845, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000053424 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:16.280 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183808 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:16.281 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:16.282 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:16.283 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148714, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 30, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T16:03:16.284 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T16:03:16.285 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:16.286 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 225, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1406470, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 225, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 225, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1406470, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:16.287 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:16.288 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:16.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:16.288 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:16.288 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T16:03:16.656 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:16 smithi023 ceph-mon[32448]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T16:03:16.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:16 smithi117 ceph-mon[36002]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T16:03:16.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:16 smithi120 ceph-mon[36826]: pgmap v316: 129 pgs: 129 active+clean; 2.3 MiB data, 502 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 127 B/s wr, 10 op/s 2024-03-28T16:03:16.949 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-28T16:03:16.949 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-28T16:03:16.949 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-28T16:03:16.949 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-28T16:03:16.949 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-28T16:03:16.949 DEBUG:tasks.check_counter:Getting stats from g 2024-03-28T16:03:16.949 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-28T16:03:17.196 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:18.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:18 smithi023 ceph-mon[32448]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 14 op/s 2024-03-28T16:03:18.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:18 smithi117 ceph-mon[36002]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 14 op/s 2024-03-28T16:03:18.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:18 smithi120 ceph-mon[36826]: pgmap v317: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 229 B/s wr, 14 op/s 2024-03-28T16:03:19.134 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T16:03:19.134 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 744, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 744, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 251577, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 235769, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.063708108, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.024141869, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.015114545, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.018604001, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 743, 2024-03-28T16:03:19.135 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.024290664, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000032692 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 448 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 1370, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 1527, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 350696, 2024-03-28T16:03:19.136 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 869993, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 7, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.078956445, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.026664628, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.018897461, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.026278615, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 1524, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.043522289, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000028557 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.137 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 434912 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 242, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 231, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1470900, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 129653, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T16:03:19.138 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.035406114, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.010041554, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.031880633, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003618573, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 230, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007887859, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000034295 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1469552, 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 128688 2024-03-28T16:03:19.139 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 1, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-MDSRank": { 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 3571, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.072953016, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000020429 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "queue_len": 0, 2024-03-28T16:03:19.140 INFO:teuthology.orchestra.run.smithi023.stdout: "complete_latency": { 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "mds": { 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "request": 0, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "reply": 0, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "reply_latency": { 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.141 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_reply": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "forward": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_commit": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_split": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_merge": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_top": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_bottom": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_pinned": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_expired": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:19.142 INFO:teuthology.orchestra.run.smithi023.stdout: "caps": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "subtrees": 1, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_hit": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_forward": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_discover": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "traverse_lock": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "load_cent": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "q": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "exported": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "exported_inodes": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "imported": 0, 2024-03-28T16:03:19.143 INFO:teuthology.orchestra.run.smithi023.stdout: "imported_inodes": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rfiles": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rbytes": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "root_rsnaps": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:19.144 INFO:teuthology.orchestra.run.smithi023.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "process_request_cap_release": 0 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_cache": { 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:19.145 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_try_discover": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_send_discover": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_created": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_enqueued": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "strays_migrated": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_started": 0, 2024-03-28T16:03:19.146 INFO:teuthology.orchestra.run.smithi023.stdout: "recovery_completed": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_flush": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ireq_inodestats": 0 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_log": { 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "evlrg": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "evadd": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "evex": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "evtrm": 0, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "ev": 7, 2024-03-28T16:03:19.147 INFO:teuthology.orchestra.run.smithi023.stdout: "evexg": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "evexd": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segadd": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segex": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segtrm": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "seg": 2, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segmjr": 2, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segexg": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "segexd": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "expos": 4194304, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "wrpos": 4196264, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "rdpos": 4196123, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "jlat": { 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.148 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "replayed": 7 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_mem": { 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "ino": 3, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "ino+": 3, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "ino-": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dir": 1, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dir+": 1, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dir-": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dn": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dn+": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "dn-": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "cap": 0, 2024-03-28T16:03:19.149 INFO:teuthology.orchestra.run.smithi023.stdout: "cap+": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "cap-": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "rss": 52940, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "heap": 215420 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_server": { 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_request": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_client_session": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "handle_peer_request": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "req_create_latency": { 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.150 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getattr_latency": { 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "req_link_latency": { 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.151 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookup_latency": { 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupino_latency": { 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.152 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupname_latency": { 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:19.153 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "req_lssnap_latency": { 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mkdir_latency": { 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.154 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.155 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mknod_latency": { 2024-03-28T16:03:19.155 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.155 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.155 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.155 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "req_mksnap_latency": { 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "req_open_latency": { 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "req_readdir_latency": { 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.156 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rename_latency": { 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.157 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmdir_latency": { 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:19.158 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setattr_latency": { 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.159 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setlayout_latency": { 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "req_setxattr_latency": { 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.160 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "req_symlink_latency": { 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "req_unlink_latency": { 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:19.161 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_sessions": { 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "session_count": 1, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "session_add": 3, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "session_remove": 2, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_open": 3, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "sessions_stale": 0, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "total_load": 0, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "average_load": 0, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "avg_session_uptime": 4122, 2024-03-28T16:03:19.162 INFO:teuthology.orchestra.run.smithi023.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:19.163 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:19.164 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 197513, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 56, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:19.165 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 188, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 8320, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 13, 2024-03-28T16:03:19.166 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "objecter": { 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_active": 0, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_laggy": 0, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send": 2111, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_send_bytes": 21, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_resend": 3, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_reply": 2111, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "op_latency": { 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2111, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.930992413, 2024-03-28T16:03:19.167 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000441019 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op_inflight": 0, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "oplen_avg": { 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 2111, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 2112 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op": 2111, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op_r": 2111, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op_w": 0, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op_rmw": 0, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "op_pg": 0, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_stat": 1363, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_create": 0, 2024-03-28T16:03:19.168 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_read": 746, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_write": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writefull": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_writesame": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_append": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_zero": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_truncate": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_delete": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_mapext": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_getxattr": 1, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_setxattr": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:19.169 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_call": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_watch": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_notify": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "osdop_other": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_active": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_send": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_resend": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "linger_ping": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_active": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_send": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "poolop_resend": 0, 2024-03-28T16:03:19.170 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_active": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_send": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "poolstat_resend": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_active": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_send": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "statfs_resend": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "command_active": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "command_send": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "command_resend": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "map_epoch": 103, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "map_full": 0, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "map_inc": 13, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_sessions": 6, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_open": 6, 2024-03-28T16:03:19.171 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_session_close": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_laggy": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_wr": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_rd": 2, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_del": 0 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "oft": { 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_objs": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_updates": 0, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "omap_total_removes": 0 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "osd-slow-ops": { 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: "slow_ops_count": 0 2024-03-28T16:03:19.172 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "purge_queue": { 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_executed": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 2352, 2024-03-28T16:03:19.173 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1846236, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 2352, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2352, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1846236, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:19.174 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2111, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 1965, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2111, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1965, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.175 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 1024, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 2111, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 2111, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 2111, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 2111, 2024-03-28T16:03:19.176 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:19.177 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-write_buf_throttle-0x55ca4d5286e0": { 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 3758096384, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 0, 2024-03-28T16:03:19.178 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:19.179 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T16:03:19.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:19 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:19.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:19 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:19.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:19 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-28T16:03:19.944 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-28T16:03:19.944 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-28T16:03:19.944 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-03-28T16:03:19.944 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:19.944 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:19.944 DEBUG:tasks.check_counter:Getting stats from j 2024-03-28T16:03:19.945 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-28T16:03:20.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:20 smithi023 ceph-mon[32448]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:20.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:20 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:20 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:20 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:20 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:20 smithi117 ceph-mon[36002]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:20.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:20 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:20 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:20 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:20 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:20 smithi120 ceph-mon[36826]: pgmap v318: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 204 B/s wr, 13 op/s 2024-03-28T16:03:20.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:20 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:20 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:20 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:20 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:20.861 INFO:teuthology.orchestra.run.smithi023.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.a/config 2024-03-28T16:03:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.671 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:21 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:21 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:21.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:21 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:22.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:22 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T16:03:22.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:22 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T16:03:22.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:22 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T16:03:22.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:22 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T16:03:22.627 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:22 smithi023 ceph-mon[32448]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout:{ 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 0, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 2, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 0, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 0, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 4, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 0, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.001187560, 2024-03-28T16:03:22.653 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.000000000, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.000291306, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.654 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 230, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 218, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 1441712, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 129760, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 2, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 1, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.034614636, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.009875330, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.031056786, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.003645090, 2024-03-28T16:03:22.655 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 218, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.007265537, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000033328 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 1441712, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 129760 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:22.656 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_messages": 2, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages": 164, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_bytes": 320, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_bytes": 183984, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_created_connections": 5, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_active_connections": 3, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_total_time": 0.010297665, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_send_time": 0.006674994, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_recv_time": 0.001648115, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 161, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.012279193, 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000076268 2024-03-28T16:03:22.657 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "msgr_send_encrypted_bytes": 183984 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "cct": { 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "total_workers": 0, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "unhealthy_workers": 0 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "mempool": { 2024-03-28T16:03:22.658 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:22.659 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:22.660 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_bytes": 148794, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_anon_items": 32, 2024-03-28T16:03:22.661 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "pgmap_items": 0, 2024-03-28T16:03:22.662 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_bytes": 928, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "mds_co_items": 2, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_1_items": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "unittest_2_items": 0 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: }, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "val": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "max": 104857600, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "get_started": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "get": 228, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "get_sum": 1406860, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "get_or_fail_success": 228, 2024-03-28T16:03:22.663 INFO:teuthology.orchestra.run.smithi023.stdout: "take": 0, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "take_sum": 0, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "put": 228, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "put_sum": 1406860, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "wait": { 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "avgcount": 0, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "sum": 0.000000000, 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: "avgtime": 0.000000000 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout: } 2024-03-28T16:03:22.664 INFO:teuthology.orchestra.run.smithi023.stdout:} 2024-03-28T16:03:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:22 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T16:03:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:22 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T16:03:22.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:22 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T16:03:22.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:22 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T16:03:22.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:22 smithi117 ceph-mon[36002]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:22 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi120 to 174.1M 2024-03-28T16:03:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:22 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi120 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-28T16:03:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:22 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi117 to 174.1M 2024-03-28T16:03:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:22 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi117 to 182633472: error parsing value: Value '182633472' is below minimum 939524096 2024-03-28T16:03:22.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:22 smithi120 ceph-mon[36826]: pgmap v319: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:23.517 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-28T16:03:23.517 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-28T16:03:23.517 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-28T16:03:23.517 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-28T16:03:23.517 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-28T16:03:23.517 DEBUG:tasks.check_counter:Getting stats from b 2024-03-28T16:03:23.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-28T16:03:23.772 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T16:03:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:24.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:24.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:24 smithi023 ceph-mon[32448]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:24.941 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:24 smithi117 ceph-mon[36002]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:25.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: pgmap v320: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-28T16:03:25.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:25.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.090 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-28T16:03:25.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-28T16:03:25.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-28T16:03:25.091 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:24 smithi120 ceph-mon[36826]: from='mgr.15546 ' entity='mgr.x' 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1510, 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1327, 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 2009479, 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 747820, 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 10, 2024-03-28T16:03:25.566 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.119821988, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.034078869, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.065874919, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.027745174, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1333, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.421853608, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000316469 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1719, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000293023, 2024-03-28T16:03:25.567 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000170 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1518288, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 446096 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1573, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 794, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 600678, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 543012, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 21, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.117016470, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.032830201, 2024-03-28T16:03:25.568 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.044798964, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.036363812, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 791, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.527806111, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000667264 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1710, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000249568, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000145 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.569 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 340336, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 227728 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 2033, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1155, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 660263, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 1758476, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 34, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-03-28T16:03:25.570 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.155045452, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.045458768, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.059723839, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.047684375, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1147, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.302035201, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000263326 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2105, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000344739, 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000163 2024-03-28T16:03:25.571 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 96016, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 37968 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 623, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.150904252, 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000242221 2024-03-28T16:03:25.572 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 79, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.003057705, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000038705 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 200, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 200, 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-28T16:03:25.573 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 200, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.155880570, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000779402 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 61, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 39, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 9, 2024-03-28T16:03:25.574 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 30, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 25, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 3, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 3, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 2, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 289, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 226, 2024-03-28T16:03:25.575 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-28T16:03:25.576 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 4, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 234, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:25.577 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 1, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 22, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 39, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:25.578 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 112, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 86, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 3, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 33 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-28T16:03:25.579 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 6, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 2, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 6, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 4, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-28T16:03:25.580 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-28T16:03:25.581 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 264, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 170, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 170, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 94, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 26, 2024-03-28T16:03:25.582 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 13, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 12, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 12, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 1, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 1, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 1, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 5055135, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 5297738, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-28T16:03:25.583 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 228, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.501748794, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002200652 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 29, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 33, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 4, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 20, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 20, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-03-28T16:03:25.584 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 39, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 44, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 5, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 3, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 24, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 21, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 56496, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 231804 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 261, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 200, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 36, 2024-03-28T16:03:25.585 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.011732429, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001466553 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 108, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.015325392, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000141901 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.586 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 30, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.005778221, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000192607 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.587 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000177274, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000177274 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.588 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:25.589 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001274476, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000127447 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-28T16:03:25.590 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-28T16:03:25.591 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.015956959, 2024-03-28T16:03:25.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001994619 2024-03-28T16:03:25.591 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.591 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-28T16:03:25.591 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 10, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.042488997, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.004248899 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.592 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001096850, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000365616 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002054390, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001027195 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.593 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012629265, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.006314632 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4, 2024-03-28T16:03:25.594 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.015589880, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.003897470 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001307496, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000653748 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:25.595 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.596 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 8, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.018618768, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002327346 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 4, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.011850173, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.002962543 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.597 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 6, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 4, 2024-03-28T16:03:25.598 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 27, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 13, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 595, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:25.599 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:25.600 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:25.601 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 359500, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 150, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 792, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 9, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 188, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:25.602 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 132621, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 1173, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 1011, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 1213649, 2024-03-28T16:03:25.603 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 31, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 1011, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1011, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 19.601613054, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.019388341 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1011, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1191 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 1011, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 1009, 2024-03-28T16:03:25.604 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 41, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 36, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 2, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 230, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 62, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 583, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-28T16:03:25.605 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 64, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-28T16:03:25.606 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-28T16:03:25.607 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 103, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 18, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 12, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 12, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 161, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 12 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-28T16:03:25.608 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 65, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 3, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 71, 2024-03-28T16:03:25.609 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 4, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 5112, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 2746703, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 5112, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:25.610 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 5112, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2746703, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:25.611 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 1011, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 1214117, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 1011, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1214117, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:25.612 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 1011, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 1011, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 1011, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1011, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.613 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 4, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 468, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 4, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2, 2024-03-28T16:03:25.614 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 468, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x559ee72f0500": { 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 264, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1103434, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 264, 2024-03-28T16:03:25.615 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 228, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1103434, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:25.616 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-28T16:03:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:25 smithi117 ceph-mon[36002]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T16:03:25.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:25 smithi117 ceph-mon[36002]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T16:03:25.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:25 smithi023 ceph-mon[32448]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T16:03:25.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:25 smithi023 ceph-mon[32448]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T16:03:26.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:25 smithi120 ceph-mon[36826]: Adjusting osd_memory_target on smithi023 to 176.2M 2024-03-28T16:03:26.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:25 smithi120 ceph-mon[36826]: Unable to set osd_memory_target on smithi023 to 184791040: error parsing value: Value '184791040' is below minimum 939524096 2024-03-28T16:03:26.363 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-03-28T16:03:26.363 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-03-28T16:03:26.363 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-28T16:03:26.363 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-28T16:03:26.363 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-28T16:03:26.363 DEBUG:tasks.check_counter:Getting stats from e 2024-03-28T16:03:26.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-28T16:03:26.628 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T16:03:26.913 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:26 smithi117 ceph-mon[36002]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:03:26.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:26 smithi023 ceph-mon[32448]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:03:27.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:26 smithi120 ceph-mon[36826]: pgmap v321: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 93 B/s wr, 9 op/s 2024-03-28T16:03:27.773 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:27 smithi117 ceph-mon[36002]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:03:27.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:27 smithi023 ceph-mon[32448]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:03:28.089 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:27 smithi120 ceph-mon[36826]: pgmap v322: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 85 B/s wr, 12 op/s 2024-03-28T16:03:28.478 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-28T16:03:28.478 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:28.478 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 2183, 2024-03-28T16:03:28.478 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2184, 2024-03-28T16:03:28.478 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 619030, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 693883, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.117774273, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.036331549, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.028743566, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.044102508, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2182, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.040002088, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000018332 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.479 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 4, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 166, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 876, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 439690, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:28.480 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.008165495, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.004710341, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.001929695, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000063333, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 163, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.010546480, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000064702 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 438992 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 253, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 237, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1491436, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 133198, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 3, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.033726870, 2024-03-28T16:03:28.482 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.008700283, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.031658604, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003618718, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 237, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.008886680, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000037496 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1491168, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 132880 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3680, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.070669258, 2024-03-28T16:03:28.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000019203 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-03-28T16:03:28.485 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 0, 2024-03-28T16:03:28.486 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 1, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:28.487 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:28.488 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:28.489 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:28.490 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-28T16:03:28.491 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 7, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4196347, 2024-03-28T16:03:28.492 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 4196206, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 7 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 3, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 3, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 1, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 1, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-03-28T16:03:28.493 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 48740, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215420 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-03-28T16:03:28.494 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.497 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.498 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.499 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.499 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.499 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-28T16:03:28.499 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-28T16:03:28.500 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:28.501 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.502 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.503 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.504 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:28.505 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 1, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 3, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 2, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-03-28T16:03:28.506 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 3584, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:28.507 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:28.508 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:28.509 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 181353, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 48, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:28.510 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 188, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 8320, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 13, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-28T16:03:28.511 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 2185, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 21, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 3, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 2185, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2185, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 4.237916621, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001939549 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2185, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2186 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.512 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 2185, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2185, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 1428, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 755, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-03-28T16:03:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 1, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:28.514 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-28T16:03:28.515 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 103, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 4, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 4, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-03-28T16:03:28.516 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-28T16:03:28.517 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 2436, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1876392, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 2436, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:28.518 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2436, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1876392, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:28.519 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2185, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2048, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2185, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2048, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.520 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2185, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2185, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2185, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2185, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.521 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:28.522 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55d1a41866e0": { 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:28.523 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:28.524 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-28T16:03:29.330 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-28T16:03:29.330 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-28T16:03:29.330 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-28T16:03:29.330 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:29.330 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:29.330 DEBUG:tasks.check_counter:Getting stats from h 2024-03-28T16:03:29.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-28T16:03:29.599 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T16:03:30.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:30 smithi120 ceph-mon[36826]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:30.622 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:30 smithi023 ceph-mon[32448]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:30.756 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:30 smithi117 ceph-mon[36002]: pgmap v323: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 358, 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 367, 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1530401, 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 218064, 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 14, 2024-03-28T16:03:31.342 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.043694899, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.011202120, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.038441660, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003880466, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 367, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.021282262, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000057989 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 146, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000029064, 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000199 2024-03-28T16:03:31.343 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1506432, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 147312 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 113, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 147, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 44328, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 183794, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.009283197, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.003359196, 2024-03-28T16:03:31.344 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.005818163, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000260113, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 146, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.019714341, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000135029 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 169, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000025044, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000148 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 131312 2024-03-28T16:03:31.345 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 314, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 634, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 93085, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 497409, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 9, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.041343208, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.020241153, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.016271434, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003042787, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 630, 2024-03-28T16:03:31.346 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.032257381, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000051202 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 782, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000141749, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000181 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 310880 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-28T16:03:31.347 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 46, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000995614, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000021643 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:31.348 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000008951, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000004475 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 6, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 3, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000680063, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000226687 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-28T16:03:31.349 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 3, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 11, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 10, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 10, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 10, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-28T16:03:31.350 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 1, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 1, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 1, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 7, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 6, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-28T16:03:31.351 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 4, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 234, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 3, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:31.352 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.353 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 1, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-28T16:03:31.354 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-28T16:03:31.355 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 3, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 3, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 2, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4195283, 2024-03-28T16:03:31.356 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002622533, 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001311266 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-28T16:03:31.357 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 13, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 13, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 11, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 12, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 1, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 10, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 10, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 1, 2024-03-28T16:03:31.358 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 1, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 49376, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223612 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 6, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 6, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 8, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.359 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000680063, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000226687 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.360 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.361 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.362 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.363 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.364 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.365 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.365 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-28T16:03:31.366 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.367 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.368 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.369 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-28T16:03:31.370 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.371 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 1, 2024-03-28T16:03:31.372 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 1, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 1, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 2, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 2, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 566, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:31.373 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:31.374 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:31.375 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:31.375 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:31.375 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:31.375 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:31.375 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:31.376 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 263281, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 92, 2024-03-28T16:03:31.377 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 176, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 2, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 188, 2024-03-28T16:03:31.378 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 50416, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 201, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.379 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 37, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 11191, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 28, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 37, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 37, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.212995789, 2024-03-28T16:03:31.380 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005756642 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 37, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 59 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 37, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 0, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 37, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-03-28T16:03:31.381 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 2, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 6, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 6, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-03-28T16:03:31.382 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-28T16:03:31.383 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 103, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-28T16:03:31.384 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 8, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 8, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 14, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 1 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.385 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:31.386 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 781, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1579943, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 781, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 781, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1579943, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.387 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 37, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 11191, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 37, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 11191, 2024-03-28T16:03:31.388 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:31.389 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 37, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 37, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 37, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 37, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:31.390 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.391 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x55b966436500": { 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 3, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 979, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 3, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 979, 2024-03-28T16:03:31.392 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:31.393 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-28T16:03:32.192 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-28T16:03:32.193 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-28T16:03:32.193 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=3 2024-03-28T16:03:32.193 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:32.193 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:32.193 DEBUG:tasks.check_counter:Getting stats from k 2024-03-28T16:03:32.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-28T16:03:32.491 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.b/config 2024-03-28T16:03:32.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:32 smithi117 ceph-mon[36002]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:32.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:32 smithi120 ceph-mon[36826]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:32.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:32 smithi023 ceph-mon[32448]: pgmap v324: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:34.247 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-03-28T16:03:34.247 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 852, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 854, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 279611, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 270937, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.060971892, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.021088555, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.014985960, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.018848671, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 852, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.024693404, 2024-03-28T16:03:34.248 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000028982 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 258, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 242, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1493374, 2024-03-28T16:03:34.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 135774, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.034943049, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.008799954, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.032962836, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.003820647, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 242, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.011630658, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000048060 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.250 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1491984, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 134752 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1292, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1456, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 329438, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 852622, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 7, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.073220385, 2024-03-28T16:03:34.251 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.025309527, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.017299315, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.024473876, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1453, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.038772953, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000026684 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 442720 2024-03-28T16:03:34.252 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3653, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.069038522, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000018899 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:34.253 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.254 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-03-28T16:03:34.255 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 1, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-03-28T16:03:34.256 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:34.257 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-03-28T16:03:34.258 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:34.259 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-03-28T16:03:34.260 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 6, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4195956, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 4195815, 2024-03-28T16:03:34.261 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 6 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 3, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 3, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-03-28T16:03:34.262 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 1, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 1, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 48160, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215420 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:34.263 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.264 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.265 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.266 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.267 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.268 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.269 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.270 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.271 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-03-28T16:03:34.272 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.273 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.274 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.275 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 2, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 2, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 1569, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:34.276 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:34.277 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:34.278 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 198337, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 59, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:34.279 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:34.280 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:34.280 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 188, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 8320, 2024-03-28T16:03:34.281 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 13, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 2144, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 21, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 3, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 2144, 2024-03-28T16:03:34.282 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2144, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 1.760965459, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000821345 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2144, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2145 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 2144, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2144, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-03-28T16:03:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 1382, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 760, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:34.284 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 1, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-03-28T16:03:34.285 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-03-28T16:03:34.286 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 103, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 13, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 6, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 6, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-03-28T16:03:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.288 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 2398, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1870737, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 2398, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2398, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1870737, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.289 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2144, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 1657, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2144, 2024-03-28T16:03:34.290 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1657, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:34.291 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2144, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2144, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2144, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2144, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:34.292 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.293 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x563861eec6e0": { 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-03-28T16:03:34.294 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-03-28T16:03:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-03-28T16:03:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-03-28T16:03:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.295 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-03-28T16:03:34.295 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-03-28T16:03:34.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:34 smithi117 ceph-mon[36002]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:34.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:34 smithi023 ceph-mon[32448]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:34.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:34 smithi120 ceph-mon[36826]: pgmap v325: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:35.034 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-28T16:03:35.034 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-28T16:03:35.034 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-03-28T16:03:35.034 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:35.034 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:35.034 DEBUG:tasks.check_counter:Getting stats from c 2024-03-28T16:03:35.034 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-28T16:03:35.286 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T16:03:36.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:36 smithi023 ceph-mon[32448]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:36.684 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:36 smithi120 ceph-mon[36826]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:36.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:36 smithi117 ceph-mon[36002]: pgmap v326: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:37.094 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 371, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 552, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 118098, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 148849, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 9, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:37.095 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.033807234, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.012993466, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.016074517, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.002952184, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 556, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.032555452, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000058552 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 857, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000143504, 2024-03-28T16:03:37.096 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000167 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 71, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 235, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 25712, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 473140, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 11, 2024-03-28T16:03:37.097 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 7, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.013951786, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.006007889, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.006354180, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000304697, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 232, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.020388836, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000087882 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:37.098 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 97, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000018319, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000188 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1248, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 444944 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 343, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 333, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1521969, 2024-03-28T16:03:37.099 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 211751, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 13, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 7, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.041285327, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.010961558, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.037086904, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.004113096, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 327, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.015697308, 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000048004 2024-03-28T16:03:37.100 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 103, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000016087, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000156 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1499600, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 138640 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.101 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 45, 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001995893, 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000044353 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:37.102 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000006536, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000003268 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.103 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 11, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-28T16:03:37.104 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 10, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 10, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 10, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-03-28T16:03:37.105 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-03-28T16:03:37.106 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-03-28T16:03:37.107 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 3, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:37.108 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-03-28T16:03:37.109 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 1, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:37.110 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-28T16:03:37.111 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-28T16:03:37.112 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 6, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 6, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-28T16:03:37.113 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 2, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4195956, 2024-03-28T16:03:37.114 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 5, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.008694062, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001738812 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 13, 2024-03-28T16:03:37.115 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 13, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 11, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 12, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 1, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 10, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 10, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-03-28T16:03:37.116 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 49740, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 223612 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 8, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-28T16:03:37.117 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:37.118 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.119 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.120 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:37.121 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.122 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.123 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.123 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-28T16:03:37.123 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.123 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.124 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.125 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.126 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.127 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.128 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.129 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:37.130 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 2, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 2, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-03-28T16:03:37.131 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 88, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:37.132 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:37.133 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 230177, 2024-03-28T16:03:37.134 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 75, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 188, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:37.135 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 50040, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 198, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 41, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 11954, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 29, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 41, 2024-03-28T16:03:37.136 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 41, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.205995324, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.005024276 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 41, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 63 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 41, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 0, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 41, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-28T16:03:37.137 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 10, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 5, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 7, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 6, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 0, 2024-03-28T16:03:37.138 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 20, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-28T16:03:37.139 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-28T16:03:37.140 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 13, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 8, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 8, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 14, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 1 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.141 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:37.142 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 781, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1578134, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 781, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 781, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1578134, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.143 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 41, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 11954, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 41, 2024-03-28T16:03:37.144 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 11954, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:37.145 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 41, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 41, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 41, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 41, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:37.146 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.147 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55b21b2e0500": { 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 6, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1652, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 6, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:37.148 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 5, 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1652, 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:37.149 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-28T16:03:37.857 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-28T16:03:37.857 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-28T16:03:37.857 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=3 2024-03-28T16:03:37.857 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:37.857 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:37.858 DEBUG:tasks.check_counter:Getting stats from f 2024-03-28T16:03:37.858 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-28T16:03:38.121 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T16:03:38.503 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:38 smithi120 ceph-mon[36826]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:38.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:38 smithi023 ceph-mon[32448]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:38.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:38 smithi117 ceph-mon[36002]: pgmap v327: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1636, 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1638, 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1586965, 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 520258, 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:39.913 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.097711084, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.028476932, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.026443391, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.033846629, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1636, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.035622001, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000021773 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.914 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 255, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 241, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1490536, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 135178, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 4, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 3, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.034031191, 2024-03-28T16:03:39.915 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.008865862, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.030256099, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.003912541, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 240, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.016717228, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000069655 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1488880, 2024-03-28T16:03:39.916 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 133840 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 767, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 928, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 190753, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 686273, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 4, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.062968861, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.022173140, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.015028159, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.019968932, 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:39.917 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 926, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.025052162, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000027054 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 444592 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-28T16:03:39.918 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3969, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.079647724, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000020067 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:39.919 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-28T16:03:39.920 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 31, 2024-03-28T16:03:39.921 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 8, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 23, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 20, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 2, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:39.922 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 6, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:39.923 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:39.924 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:39.926 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-28T16:03:39.927 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-28T16:03:39.928 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 264, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 13, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 5055135, 2024-03-28T16:03:39.929 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 5297738, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 5297602, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 264 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 20, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 24, 2024-03-28T16:03:39.930 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 4, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 11, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 11, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 31, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 31, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-03-28T16:03:39.931 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 48700, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215420 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-28T16:03:39.932 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.933 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-28T16:03:39.934 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.935 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.936 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.937 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.937 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-28T16:03:39.937 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.937 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.937 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.938 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-28T16:03:39.939 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.940 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.941 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.942 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-28T16:03:39.943 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.944 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 2, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 6, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 4, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 4, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 3588, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-28T16:03:39.945 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:39.946 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:39.947 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 189726, 2024-03-28T16:03:39.948 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 60, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 188, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-28T16:03:39.949 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 91949, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 760, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 2400, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 26, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 4, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 2400, 2024-03-28T16:03:39.950 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2400, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2.539942555, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001058309 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2400, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2401 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 2400, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 2400, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-28T16:03:39.951 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 1518, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 879, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 2, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-03-28T16:03:39.952 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-28T16:03:39.953 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 13, 2024-03-28T16:03:39.954 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 5, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 5, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-28T16:03:39.955 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:39.956 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2654, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 3013534, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2654, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2654, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 3013534, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.957 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2400, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 1103444, 2024-03-28T16:03:39.958 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2400, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1103444, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:39.959 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2400, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2400, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2400, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2400, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:39.960 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.961 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55eac2f966e0": { 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:39.962 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:39.963 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-28T16:03:40.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:40 smithi023 ceph-mon[32448]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:40.734 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:40 smithi120 ceph-mon[36826]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:40.764 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-28T16:03:40.764 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-28T16:03:40.764 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-03-28T16:03:40.764 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:40.764 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:40.764 DEBUG:tasks.check_counter:Getting stats from i 2024-03-28T16:03:40.765 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-28T16:03:40.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:40 smithi117 ceph-mon[36002]: pgmap v328: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:41.014 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T16:03:42.570 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:42 smithi120 ceph-mon[36826]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:42.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:42 smithi023 ceph-mon[32448]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:42.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:42 smithi117 ceph-mon[36002]: pgmap v329: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:42.847 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-28T16:03:42.847 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:42.847 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 88, 2024-03-28T16:03:42.847 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 219, 2024-03-28T16:03:42.847 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 29884, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 394149, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 17, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.015146641, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.005317896, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.007131211, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.000557081, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 216, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.032118418, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000148696 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.848 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 110, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000033054, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000300 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 2096, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 316016 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 375, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 594, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 107211, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 361469, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 11, 2024-03-28T16:03:42.849 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 6, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.037237829, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.014291636, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.018973066, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.003014350, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 590, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.027914057, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000047311 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 897, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000150288, 2024-03-28T16:03:42.850 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000167 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 2656, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 137264 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 391, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 368, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1550477, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 187517, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 7, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.042662822, 2024-03-28T16:03:42.851 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.011502050, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.039716161, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.003951393, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 370, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.024289087, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000065646 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 180, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000036144, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000200 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1507296, 2024-03-28T16:03:42.852 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 137504 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 44, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.001929065, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000043842 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.853 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000006389, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000003194 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 3, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 1, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-03-28T16:03:42.854 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000177058, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000177058 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 2, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 11, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 11, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 11, 2024-03-28T16:03:42.855 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 10, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 2, 2024-03-28T16:03:42.856 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 5, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 2, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 1, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-03-28T16:03:42.857 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 4, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 234, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 3, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:42.858 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:42.859 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 2 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 2, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:42.860 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-28T16:03:42.861 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 7, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 7, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 2, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-28T16:03:42.862 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4196347, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 0, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 6, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.009668017, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.001611336 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 0 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-28T16:03:42.863 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 14, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 14, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 12, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 13, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 1, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 11, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 11, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 1, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 1, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 49532, 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 223612 2024-03-28T16:03:42.864 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 4, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 3, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 16, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000177058, 2024-03-28T16:03:42.865 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000177058 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-28T16:03:42.866 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.867 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.868 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.869 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.870 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.871 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:42.872 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.873 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.874 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:42.875 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.876 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.877 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-28T16:03:42.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.877 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.877 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.878 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 3, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 2, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 1, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 1, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 578, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.879 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:42.880 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:42.881 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:42.882 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 263507, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 95, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:42.883 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 188, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-28T16:03:42.884 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 54184, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 213, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 44, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 12525, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 28, 2024-03-28T16:03:42.885 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 44, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 44, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.214995121, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.004886252 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 44, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 66 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 44, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 0, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 44, 2024-03-28T16:03:42.886 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 10, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 6, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 9, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 6, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:42.887 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 20, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-28T16:03:42.888 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-28T16:03:42.889 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 13, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 9, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 9, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 14, 2024-03-28T16:03:42.890 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 0, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 1 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 1, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.895 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:42.896 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 850, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1593101, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 850, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 850, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1593101, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.897 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 44, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 12525, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 44, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 12525, 2024-03-28T16:03:42.898 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 44, 2024-03-28T16:03:42.899 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 44, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 44, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 44, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:42.900 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x55a1c1dc6500": { 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:42.901 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 7, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 2043, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 7, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 6, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2043, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:42.902 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.903 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:42.903 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-28T16:03:43.525 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-28T16:03:43.525 INFO:tasks.check_counter:Daemon mds.i mds.imported=0 2024-03-28T16:03:43.525 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=3 2024-03-28T16:03:43.525 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:43.526 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:43.526 DEBUG:tasks.check_counter:Getting stats from l 2024-03-28T16:03:43.526 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid a96af96e-ed19-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-28T16:03:43.801 INFO:teuthology.orchestra.run.smithi120.stderr:Inferring config /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/mon.c/config 2024-03-28T16:03:44.589 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:44 smithi120 ceph-mon[36826]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:44.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:44 smithi023 ceph-mon[32448]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:44.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:44 smithi117 ceph-mon[36002]: pgmap v330: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout:{ 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-0": { 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 771, 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 773, 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 203588, 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 245242, 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 6, 2024-03-28T16:03:45.581 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.044894630, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.013851828, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.012021544, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.014117433, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 771, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.019149010, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000024836 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.582 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 0 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-1": { 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 588, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 756, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 144498, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 632801, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 7, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 5, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.045397248, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.015089166, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.010082317, 2024-03-28T16:03:45.583 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.015829637, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 753, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.026082682, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000034638 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 446448 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.584 INFO:teuthology.orchestra.run.smithi120.stdout: "AsyncMessenger::Worker-2": { 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_messages": 1025, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages": 1009, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_bytes": 1750448, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_bytes": 380836, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_created_connections": 3, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_active_connections": 2, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_total_time": 0.090547408, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_send_time": 0.025701267, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_recv_time": 0.048864128, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_running_fast_dispatch_time": 0.022036135, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_messages_queue_lat": { 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 1009, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.029952099, 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000029684 2024-03-28T16:03:45.585 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_handle_ack_lat": { 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_recv_encrypted_bytes": 1493072, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "msgr_send_encrypted_bytes": 137248 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "cct": { 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "total_workers": 1, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "unhealthy_workers": 0 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.586 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-MDSRank": { 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 3603, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.072247092, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000020051 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "finisher-PurgeQueue": { 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "queue_len": 0, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "complete_latency": { 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.587 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "mds": { 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "request": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "reply": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "reply_latency": { 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_reply": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "forward": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_complete": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_fetch_keys": 0, 2024-03-28T16:03:45.588 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_commit": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_split": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_merge": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_top": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_bottom": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pin_tail": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_pinned": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_expired": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "inodes_with_caps": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "caps": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "subtrees": 1, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse": 0, 2024-03-28T16:03:45.589 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_hit": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_forward": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_discover": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_dir_fetch": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_remote_ino": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "traverse_lock": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "load_cent": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "q": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "exported": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "exported_inodes": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "imported": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "imported_inodes": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_dir_fetch": 0, 2024-03-28T16:03:45.590 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_backtrace_fetch": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "openino_peer_discover": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rfiles": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rbytes": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "root_rsnaps": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_fetch": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_set_tag": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_backtrace_repaired": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_inotable_repaired": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_inodes": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dir_base_inodes": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_dirfrag_rstats": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "scrub_file_inodes": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_inode_file_caps": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_revoke": 0, 2024-03-28T16:03:45.591 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_grant": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_trunc": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_caps_dirty": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_cap_release": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "process_request_cap_release": 0 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_cache": { 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_update_receipt": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_try_discover": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_send_discover": 0, 2024-03-28T16:03:45.592 INFO:teuthology.orchestra.run.smithi120.stdout: "dir_handle_discover": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_delayed": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_strays_enqueuing": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_created": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_enqueued": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_reintegrated": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "strays_migrated": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_processing": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_enqueued": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "num_recovering_prioritized": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_started": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "recovery_completed": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_enqueue_scrub": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_exportdir": 0, 2024-03-28T16:03:45.593 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_flush": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragmentdir": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_fragstats": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "ireq_inodestats": 0 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_log": { 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evlrg": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evadd": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evex": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evtrm": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "ev": 3, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evexg": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "evexd": 0, 2024-03-28T16:03:45.594 INFO:teuthology.orchestra.run.smithi120.stdout: "segadd": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "segex": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "segtrm": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "seg": 2, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "segmjr": 2, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "segexg": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "segexd": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "expos": 4194304, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "wrpos": 4195283, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "rdpos": 4194892, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "jlat": { 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.595 INFO:teuthology.orchestra.run.smithi120.stdout: "replayed": 3 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_mem": { 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "ino": 3, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "ino+": 3, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "ino-": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dir": 1, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dir+": 1, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dir-": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dn": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dn+": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "dn-": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "cap": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "cap+": 0, 2024-03-28T16:03:45.596 INFO:teuthology.orchestra.run.smithi120.stdout: "cap-": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "rss": 47388, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "heap": 215420 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_server": { 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_client_request": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "dispatch_server_request": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_request": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_client_session": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "handle_peer_request": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "req_create_latency": { 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.597 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getattr_latency": { 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getfilelock_latency": { 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "req_link_latency": { 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.598 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookup_latency": { 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookuphash_latency": { 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupino_latency": { 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.599 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupname_latency": { 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupparent_latency": { 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lookupsnap_latency": { 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.600 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "req_lssnap_latency": { 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mkdir_latency": { 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.601 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mknod_latency": { 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "req_mksnap_latency": { 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.602 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "req_open_latency": { 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "req_readdir_latency": { 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rename_latency": { 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.603 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "req_renamesnap_latency": { 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "req_snapdiff_latency": { 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmdir_latency": { 2024-03-28T16:03:45.604 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmsnap_latency": { 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "req_rmxattr_latency": { 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.605 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setattr_latency": { 2024-03-28T16:03:45.606 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.606 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.606 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setdirlayout_latency": { 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setfilelock_latency": { 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setlayout_latency": { 2024-03-28T16:03:45.607 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "req_setxattr_latency": { 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "req_symlink_latency": { 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.608 INFO:teuthology.orchestra.run.smithi120.stdout: "req_unlink_latency": { 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_revoke_eviction": 0, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "cap_acquisition_throttle": 0, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "req_getvxattr_latency": { 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_sessions": { 2024-03-28T16:03:45.609 INFO:teuthology.orchestra.run.smithi120.stdout: "session_count": 1, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "session_add": 1, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "session_remove": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_open": 1, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "sessions_stale": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "total_load": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "average_load": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "avg_session_uptime": 2114, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "mdthresh_evicted": 0 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "mempool": { 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_bytes": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "bloom_filter_items": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_bytes": 0, 2024-03-28T16:03:45.610 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_alloc_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_data_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_onode_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_meta_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_other_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_cache_buffer_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_extent_items": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_bytes": 0, 2024-03-28T16:03:45.611 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_blob_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_shared_blob_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_inline_bl_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_fsck_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_txc_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_deferred_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluestore_writing_items": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_bytes": 0, 2024-03-28T16:03:45.612 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_reader_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "bluefs_file_writer_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_bytes": 190257, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_anon_items": 55, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "buffer_meta_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_mapbl_items": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_bytes": 0, 2024-03-28T16:03:45.613 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_pglog_items": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_bytes": 9800, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_items": 188, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_bytes": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "osdmap_mapping_items": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_bytes": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "pgmap_items": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_bytes": 8320, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "mds_co_items": 13, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_bytes": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_1_items": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_bytes": 0, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "unittest_2_items": 0 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.614 INFO:teuthology.orchestra.run.smithi120.stdout: "objecter": { 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_active": 0, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_laggy": 0, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send": 2123, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_send_bytes": 21, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_resend": 3, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_reply": 2123, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_latency": { 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2123, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.847980957, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000399425 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "op_inflight": 0, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "oplen_avg": { 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 2123, 2024-03-28T16:03:45.615 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 2124 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "op": 2123, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "op_r": 2123, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "op_w": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "op_rmw": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "op_pg": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_stat": 1350, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_create": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_read": 771, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_write": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writefull": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_writesame": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_append": 0, 2024-03-28T16:03:45.616 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_zero": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_truncate": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_delete": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_mapext": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_sparse_read": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_clonerange": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_getxattr": 1, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_setxattr": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_cmpxattr": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_rmxattr": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_resetxattrs": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_call": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_watch": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_notify": 0, 2024-03-28T16:03:45.617 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_src_cmpxattr": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_pgls_filter": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "osdop_other": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_active": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_send": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_resend": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "linger_ping": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_active": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_send": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolop_resend": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_active": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_send": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "poolstat_resend": 0, 2024-03-28T16:03:45.618 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_active": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_send": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "statfs_resend": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "command_active": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "command_send": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "command_resend": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "map_epoch": 103, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "map_full": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "map_inc": 13, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_sessions": 5, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_open": 5, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_session_close": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "osd_laggy": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_wr": 0, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_rd": 2, 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_del": 0 2024-03-28T16:03:45.619 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "oft": { 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_objs": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_kv_pairs": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_updates": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "omap_total_removes": 0 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "osd-slow-ops": { 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "slow_ops_count": 0 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "purge_queue": { 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_ops_high_water": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executing_high_water": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed_ops": 0, 2024-03-28T16:03:45.620 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_executed": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "pq_item_in_journal": 0 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 2380, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 1868260, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 2380, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2380, 2024-03-28T16:03:45.621 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 1868260, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_bytes": { 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 104857600, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:45.622 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2123, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 984, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2123, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 984, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-objecter_ops": { 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:45.623 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 1024, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 2123, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 2123, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 2123, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 2123, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle": { 2024-03-28T16:03:45.624 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.625 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: }, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "throttle-write_buf_throttle-0x561d2f2326e0": { 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "val": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "max": 3758096384, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "get_started": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "get": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "get_sum": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_fail": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "get_or_fail_success": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "take": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "take_sum": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "put": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "put_sum": 0, 2024-03-28T16:03:45.626 INFO:teuthology.orchestra.run.smithi120.stdout: "wait": { 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout: "avgcount": 0, 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout: "sum": 0.000000000, 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout: "avgtime": 0.000000000 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout: } 2024-03-28T16:03:45.627 INFO:teuthology.orchestra.run.smithi120.stdout:} 2024-03-28T16:03:46.472 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-28T16:03:46.472 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-28T16:03:46.472 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-03-28T16:03:46.472 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-03-28T16:03:46.472 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-03-28T16:03:46.472 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-28T16:03:46.483 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-28T16:03:46.483 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T16:03:46.483 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T16:03:46.500 DEBUG:teuthology.orchestra.run.smithi023:> set -ex 2024-03-28T16:03:46.501 DEBUG:teuthology.orchestra.run.smithi023:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-28T16:03:46.557 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:46.670 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:46 smithi023 ceph-mon[32448]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:46.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 -- 172.21.15.23:0/3139623551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 msgr2=0x7f0018107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:46.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3139623551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f0018107a50 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0010009cd0 tx=0x7f001002c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 -- 172.21.15.23:0/3139623551 shutdown_connections 2024-03-28T16:03:46.712 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3139623551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018102020 0x7f001810e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3139623551 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018101660 0x7f0018101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3139623551 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f0018107a50 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 -- 172.21.15.23:0/3139623551 >> 172.21.15.23:0/3139623551 conn(0x7f00180fd110 msgr2=0x7f00180ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.709+0000 7f001ec9a640 1 -- 172.21.15.23:0/3139623551 shutdown_connections 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.710+0000 7f001ec9a640 1 -- 172.21.15.23:0/3139623551 wait complete. 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.710+0000 7f001ec9a640 1 Processor -- start 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.710+0000 7f001ec9a640 1 -- start start 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 0x7f0018197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018102020 0x7f0018198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f001819c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018111a80 con 0x7f0018107670 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018111900 con 0x7f0018101660 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0018111c00 con 0x7f0018102020 2024-03-28T16:03:46.713 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001d237640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f001819c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 0x7f0018197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001d237640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f001819c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59578/0 (socket says 172.21.15.23:59578) 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001d237640 1 -- 172.21.15.23:0/3725420669 learned_addr learned my addr 172.21.15.23:0/3725420669 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f0017fff640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018102020 0x7f0018198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 -- 172.21.15.23:0/3725420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018102020 msgr2=0x7f0018198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018102020 0x7f0018198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 -- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 msgr2=0x7f001819c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f001819c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 -- 172.21.15.23:0/3725420669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001819cfb0 con 0x7f0018101660 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ca36640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 0x7f0018197fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f001002c700 tx=0x7f0010032c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010041020 con 0x7f0018101660 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0010034cd0 con 0x7f0018101660 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001819d240 con 0x7f0018101660 2024-03-28T16:03:46.714 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.711+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010039040 con 0x7f0018101660 2024-03-28T16:03:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.712+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00181a4b30 con 0x7f0018101660 2024-03-28T16:03:46.715 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.713+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0018102da0 con 0x7f0018101660 2024-03-28T16:03:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.714+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0010044020 con 0x7f0018101660 2024-03-28T16:03:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.714+0000 7f0015ffb640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 0x7f00000a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.714+0000 7f0017fff640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 0x7f00000a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.714+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f00100f10e0 con 0x7f0018101660 2024-03-28T16:03:46.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.715+0000 7f0017fff640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 0x7f00000a4e30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00180ff7b0 tx=0x7f000c005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:46.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.718+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f001004b020 con 0x7f0018101660 2024-03-28T16:03:46.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:46 smithi117 ceph-mon[36002]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:46.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:46 smithi120 ceph-mon[36826]: pgmap v331: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:46.950 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.950+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0018109bd0 con 0x7f0018101660 2024-03-28T16:03:46.952 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.951+0000 7f0015ffb640 1 -- 172.21.15.23:0/3725420669 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25476 (secure 0 0 0) 0x7f00100bdb30 con 0x7f0018101660 2024-03-28T16:03:46.952 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:46.952 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T16:03:14.372495+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1859625110":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/3007922889":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/4155829879":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/421542303":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 msgr2=0x7f00000a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 0x7f00000a4e30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f00180ff7b0 tx=0x7f000c005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 msgr2=0x7f0018197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 0x7f0018197fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f001002c700 tx=0x7f0010032c90 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 shutdown_connections 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f00000a2970 0x7f00000a4e30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0018107670 0x7f001819c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0018102020 0x7f0018198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 --2- 172.21.15.23:0/3725420669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0018101660 0x7f0018197fc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 >> 172.21.15.23:0/3725420669 conn(0x7f00180fd110 msgr2=0x7f00180fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:46.955 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 shutdown_connections 2024-03-28T16:03:46.956 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:46.954+0000 7f001ec9a640 1 -- 172.21.15.23:0/3725420669 wait complete. 2024-03-28T16:03:46.965 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-28T15:45:58.632126+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-28T15:50:30.115107+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-28T15:50:30.959648+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-28T15:50:34.679185+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '102', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 102, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-28T16:03:46.966 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-28T16:03:47.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.071+0000 7fc07c33b640 1 -- 172.21.15.23:0/2385381495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc074068880 msgr2=0x7fc074068c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.071+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2385381495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc074068880 0x7fc074068c60 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc070009900 tx=0x7fc07002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 -- 172.21.15.23:0/2385381495 shutdown_connections 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2385381495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc07410e040 0x7fc074110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2385381495 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 0x7fc07410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2385381495 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc074068880 0x7fc074068c60 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 -- 172.21.15.23:0/2385381495 >> 172.21.15.23:0/2385381495 conn(0x7fc0740776a0 msgr2=0x7fc074077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 -- 172.21.15.23:0/2385381495 shutdown_connections 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 -- 172.21.15.23:0/2385381495 wait complete. 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 Processor -- start 2024-03-28T16:03:47.073 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.072+0000 7fc07c33b640 1 -- start start 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 0x7fc07419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 0x7fc07419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074113c50 con 0x7fc07410e040 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074113ad0 con 0x7fc074068880 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc074113dd0 con 0x7fc0740691a0 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a0d7640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 0x7fc07419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59600/0 (socket says 172.21.15.23:59600) 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 -- 172.21.15.23:0/2688097437 learned_addr learned my addr 172.21.15.23:0/2688097437 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:47.076 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc0798d6640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 0x7fc07419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 -- 172.21.15.23:0/2688097437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 msgr2=0x7fc07419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 0x7fc07419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 -- 172.21.15.23:0/2688097437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 msgr2=0x7fc07419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 0x7fc07419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 -- 172.21.15.23:0/2688097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07419f0d0 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a0d7640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 0x7fc07419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07a8d8640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00eb40 tx=0x7fc06c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c004b20 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc07419f3c0 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc06c004cc0 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.073+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc06c010850 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.074+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0741a6c50 con 0x7fc07410e040 2024-03-28T16:03:47.077 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.075+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc074068490 con 0x7fc07410e040 2024-03-28T16:03:47.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.076+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc06c005020 con 0x7fc07410e040 2024-03-28T16:03:47.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.076+0000 7fc06b7fe640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 0x7fc0600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.076+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7fc06c0cf9d0 con 0x7fc07410e040 2024-03-28T16:03:47.078 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.076+0000 7fc07a0d7640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 0x7fc0600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.079 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.077+0000 7fc07a0d7640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 0x7fc0600a4e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc070000c00 tx=0x7fc07002df90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.081 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.081+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc06c09c310 con 0x7fc07410e040 2024-03-28T16:03:47.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.264+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc07402d070 con 0x7fc07410e040 2024-03-28T16:03:47.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.264+0000 7fc06b7fe640 1 -- 172.21.15.23:0/2688097437 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc06c09bc10 con 0x7fc07410e040 2024-03-28T16:03:47.265 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 1 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 msgr2=0x7fc0600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 0x7fc0600a4e30 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc070000c00 tx=0x7fc07002df90 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 msgr2=0x7fc07419e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00eb40 tx=0x7fc06c00c2d0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 shutdown_connections 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7fc0600a2970 0x7fc0600a4e30 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7fc07410e040 0x7fc07419e9a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7fc0740691a0 0x7fc07419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 --2- 172.21.15.23:0/2688097437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc074068880 0x7fc07419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 >> 172.21.15.23:0/2688097437 conn(0x7fc0740776a0 msgr2=0x7fc07410f550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 shutdown_connections 2024-03-28T16:03:47.267 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.266+0000 7fc07c33b640 1 -- 172.21.15.23:0/2688097437 wait complete. 2024-03-28T16:03:47.278 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-28T16:03:47.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.382+0000 7f0aa291b640 1 -- 172.21.15.23:0/2525521906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 msgr2=0x7f0a9c1102e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 --2- 172.21.15.23:0/2525521906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c1102e0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009ca0 tx=0x7f0a9402c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 -- 172.21.15.23:0/2525521906 shutdown_connections 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 --2- 172.21.15.23:0/2525521906 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c1102e0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 --2- 172.21.15.23:0/2525521906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c06a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 --2- 172.21.15.23:0/2525521906 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 0x7f0a9c069c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 -- 172.21.15.23:0/2525521906 >> 172.21.15.23:0/2525521906 conn(0x7f0a9c077580 msgr2=0x7f0a9c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 -- 172.21.15.23:0/2525521906 shutdown_connections 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 -- 172.21.15.23:0/2525521906 wait complete. 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 Processor -- start 2024-03-28T16:03:47.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.383+0000 7f0aa291b640 1 -- start start 2024-03-28T16:03:47.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 0x7f0a9c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c113c20 con 0x7f0a9c10def0 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c113aa0 con 0x7f0a9c06a150 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa291b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a9c113da0 con 0x7f0a9c102560 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0a9bfff640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59622/0 (socket says 172.21.15.23:59622) 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 -- 172.21.15.23:0/4209309522 learned_addr learned my addr 172.21.15.23:0/4209309522 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0a9b7fe640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 0x7f0a9c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 -- 172.21.15.23:0/4209309522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 msgr2=0x7f0a9c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 0x7f0a9c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 -- 172.21.15.23:0/4209309522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 msgr2=0x7f0a9c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 -- 172.21.15.23:0/4209309522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c19f0b0 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0a9bfff640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.384+0000 7f0aa0eb8640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0a940348f0 tx=0x7f0a94032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.385+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94041020 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.385+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a94032e80 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.385+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c19f340 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.385+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a94039040 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.385+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c1a6c30 con 0x7f0a9c10def0 2024-03-28T16:03:47.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.386+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a9c100b80 con 0x7f0a9c10def0 2024-03-28T16:03:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.387+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0a94044020 con 0x7f0a9c10def0 2024-03-28T16:03:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.388+0000 7f0a997fa640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 0x7f0a840a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.388+0000 7f0a9bfff640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 0x7f0a840a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.388+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f0a940f0c90 con 0x7f0a9c10def0 2024-03-28T16:03:47.389 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.388+0000 7f0a9bfff640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 0x7f0a840a4eb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0095f0 tx=0x7f0a8c005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.396 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.396+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a9404b020 con 0x7f0a9c10def0 2024-03-28T16:03:47.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.594+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a9c19b4b0 con 0x7f0a9c10def0 2024-03-28T16:03:47.595 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.594+0000 7f0a997fa640 1 -- 172.21.15.23:0/4209309522 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0a940bd6e0 con 0x7f0a9c10def0 2024-03-28T16:03:47.595 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T16:03:47.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 msgr2=0x7f0a840a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 0x7f0a840a4eb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0095f0 tx=0x7f0a8c005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:47.597 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 msgr2=0x7f0a9c19e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f0a940348f0 tx=0x7f0a94032c90 comp rx=0 tx=0).stop 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 shutdown_connections 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f0a840a29f0 0x7f0a840a4eb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f0a9c10def0 0x7f0a9c19e980 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f0a9c102560 0x7f0a9c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 --2- 172.21.15.23:0/4209309522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a9c06a150 0x7f0a9c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 >> 172.21.15.23:0/4209309522 conn(0x7f0a9c077580 msgr2=0x7f0a9c10ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 shutdown_connections 2024-03-28T16:03:47.598 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.597+0000 7f0aa291b640 1 -- 172.21.15.23:0/4209309522 wait complete. 2024-03-28T16:03:47.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3725420669' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:47.607 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:47 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2688097437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:47.608 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-28T16:03:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- 172.21.15.23:0/2828644149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4107630 msgr2=0x7f39b4107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 --2- 172.21.15.23:0/2828644149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4107630 0x7f39b4107a10 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f39a8009930 tx=0x7f39a802bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- 172.21.15.23:0/2828644149 shutdown_connections 2024-03-28T16:03:47.716 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 --2- 172.21.15.23:0/2828644149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4101fe0 0x7f39b410e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 --2- 172.21.15.23:0/2828644149 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101620 0x7f39b4101aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 --2- 172.21.15.23:0/2828644149 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4107630 0x7f39b4107a10 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- 172.21.15.23:0/2828644149 >> 172.21.15.23:0/2828644149 conn(0x7f39b40fd130 msgr2=0x7f39b40ff550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- 172.21.15.23:0/2828644149 shutdown_connections 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- 172.21.15.23:0/2828644149 wait complete. 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 Processor -- start 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.714+0000 7f39bab45640 1 -- start start 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 0x7f39b41984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4107630 0x7f39b419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4111a40 con 0x7f39b4101620 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b41118c0 con 0x7f39b4107630 2024-03-28T16:03:47.717 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4111bc0 con 0x7f39b4101fe0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b88e1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b88e1640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59650/0 (socket says 172.21.15.23:59650) 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b88e1640 1 -- 172.21.15.23:0/673074223 learned_addr learned my addr 172.21.15.23:0/673074223 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b90e2640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4107630 0x7f39b419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 0x7f39b41984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 -- 172.21.15.23:0/673074223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4107630 msgr2=0x7f39b419c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4107630 0x7f39b419c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 -- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 msgr2=0x7f39b4197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 -- 172.21.15.23:0/673074223 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b419cfa0 con 0x7f39b4101fe0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b88e1640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b3fff640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 0x7f39b41984f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e9f0 tx=0x7f39a000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.718 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000ccc0 con 0x7f39b4101fe0 2024-03-28T16:03:47.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39a00035f0 con 0x7f39b4101fe0 2024-03-28T16:03:47.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b419d290 con 0x7f39b4101fe0 2024-03-28T16:03:47.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.715+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000ce60 con 0x7f39b4101fe0 2024-03-28T16:03:47.719 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.716+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b41a4b20 con 0x7f39b4101fe0 2024-03-28T16:03:47.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.717+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b4068490 con 0x7f39b4101fe0 2024-03-28T16:03:47.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.718+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f39a00037a0 con 0x7f39b4101fe0 2024-03-28T16:03:47.720 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.719+0000 7f39b1ffb640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 0x7f39a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:47.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.719+0000 7f39b88e1640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 0x7f39a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:47.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.719+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f39a00cf4e0 con 0x7f39b4101fe0 2024-03-28T16:03:47.721 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.719+0000 7f39b88e1640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 0x7f39a40a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000c00 tx=0x7f39a80058d0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:47.727 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.727+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f39a009be20 con 0x7f39b4101fe0 2024-03-28T16:03:47.819 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3725420669' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:47.820 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:47 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2688097437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3725420669' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:47.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:47 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2688097437' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-28T16:03:47.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.910+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39b4109bf0 con 0x7f39b4101fe0 2024-03-28T16:03:47.910 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.910+0000 7f39b1ffb640 1 -- 172.21.15.23:0/673074223 <== mon.1 v2:172.21.15.120:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39a009b720 con 0x7f39b4101fe0 2024-03-28T16:03:47.911 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 64 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 msgr2=0x7f39a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 0x7f39a40a4e30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000c00 tx=0x7f39a80058d0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 msgr2=0x7f39b41984f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 0x7f39b41984f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f39a000e9f0 tx=0x7f39a000edf0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 shutdown_connections 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f39a40a2970 0x7f39a40a4e30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b4107630 0x7f39b419c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.913 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f39b4101fe0 0x7f39b41984f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 --2- 172.21.15.23:0/673074223 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f39b4101620 0x7f39b4197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:47.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.912+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 >> 172.21.15.23:0/673074223 conn(0x7f39b40fd130 msgr2=0x7f39b40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:47.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.913+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 shutdown_connections 2024-03-28T16:03:47.914 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:47.913+0000 7f39bab45640 1 -- 172.21.15.23:0/673074223 wait complete. 2024-03-28T16:03:47.924 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-28T16:03:48.053 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 -- 172.21.15.23:0/3157292291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c0688a0 msgr2=0x7ff36c068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 --2- 172.21.15.23:0/3157292291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c0688a0 0x7ff36c068d20 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009930 tx=0x7ff35c02bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 -- 172.21.15.23:0/3157292291 shutdown_connections 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 --2- 172.21.15.23:0/3157292291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c069260 0x7ff36c10e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 --2- 172.21.15.23:0/3157292291 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c0688a0 0x7ff36c068d20 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 --2- 172.21.15.23:0/3157292291 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c1076f0 0x7ff36c107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 -- 172.21.15.23:0/3157292291 >> 172.21.15.23:0/3157292291 conn(0x7ff36c0776a0 msgr2=0x7ff36c077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 -- 172.21.15.23:0/3157292291 shutdown_connections 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.051+0000 7ff3719f3640 1 -- 172.21.15.23:0/3157292291 wait complete. 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 Processor -- start 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 -- start start 2024-03-28T16:03:48.054 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c0688a0 0x7ff36c197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c1076f0 0x7ff36c19c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c111a40 con 0x7ff36c069260 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c1118c0 con 0x7ff36c1076f0 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff3719f3640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c111bc0 con 0x7ff36c0688a0 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff36a7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff36a7fc640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59674/0 (socket says 172.21.15.23:59674) 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.052+0000 7ff36a7fc640 1 -- 172.21.15.23:0/1155633198 learned_addr learned my addr 172.21.15.23:0/1155633198 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36b7fe640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c1076f0 0x7ff36c19c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36affd640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c0688a0 0x7ff36c197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 -- 172.21.15.23:0/1155633198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c0688a0 msgr2=0x7ff36c197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c0688a0 0x7ff36c197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 -- 172.21.15.23:0/1155633198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c1076f0 msgr2=0x7ff36c19c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.055 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c1076f0 0x7ff36c19c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 -- 172.21.15.23:0/1155633198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c19cf10 con 0x7ff36c069260 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff36a7fc640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009230 tx=0x7ff35c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c041070 con 0x7ff36c069260 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff35c0054e0 con 0x7ff36c069260 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c19d1a0 con 0x7ff36c069260 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c003c90 con 0x7ff36c069260 2024-03-28T16:03:48.056 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.053+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c1a4a90 con 0x7ff36c069260 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.054+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c199270 con 0x7ff36c069260 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.056+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff35c044070 con 0x7ff36c069260 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.056+0000 7ff3709f1640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 0x7ff3540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.057+0000 7ff36affd640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 0x7ff3540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.057+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7ff35c0f0af0 con 0x7ff36c069260 2024-03-28T16:03:48.057 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.057+0000 7ff36affd640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 0x7ff3540a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff360009bd0 tx=0x7ff360005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:48.061 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.061+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff35c0bcef0 con 0x7ff36c069260 2024-03-28T16:03:48.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.261+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff36c02d070 con 0x7ff36c069260 2024-03-28T16:03:48.261 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.261+0000 7ff3709f1640 1 -- 172.21.15.23:0/1155633198 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff35c0c0f90 con 0x7ff36c069260 2024-03-28T16:03:48.262 INFO:teuthology.orchestra.run.smithi023.stdout:pg_num: 32 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 msgr2=0x7ff3540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 0x7ff3540a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff360009bd0 tx=0x7ff360005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 msgr2=0x7ff36c198460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009230 tx=0x7ff35c004270 comp rx=0 tx=0).stop 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 shutdown_connections 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7ff3540a2970 0x7ff3540a4e30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff36c1076f0 0x7ff36c19c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.264 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7ff36c069260 0x7ff36c198460 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 --2- 172.21.15.23:0/1155633198 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7ff36c0688a0 0x7ff36c197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.263+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 >> 172.21.15.23:0/1155633198 conn(0x7ff36c0776a0 msgr2=0x7ff36c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:48.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.264+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 shutdown_connections 2024-03-28T16:03:48.265 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.264+0000 7ff3719f3640 1 -- 172.21.15.23:0/1155633198 wait complete. 2024-03-28T16:03:48.275 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-28T16:03:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 -- 172.21.15.23:0/77184658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a40688a0 msgr2=0x7f43a4068d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 --2- 172.21.15.23:0/77184658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a40688a0 0x7f43a4068d20 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4394009960 tx=0x7f439402bc20 comp rx=0 tx=0).stop 2024-03-28T16:03:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 -- 172.21.15.23:0/77184658 shutdown_connections 2024-03-28T16:03:48.383 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 --2- 172.21.15.23:0/77184658 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a4069260 0x7f43a410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 --2- 172.21.15.23:0/77184658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a40688a0 0x7f43a4068d20 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 --2- 172.21.15.23:0/77184658 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a41076f0 0x7f43a4107ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 -- 172.21.15.23:0/77184658 >> 172.21.15.23:0/77184658 conn(0x7f43a40776a0 msgr2=0x7f43a4077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 -- 172.21.15.23:0/77184658 shutdown_connections 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.380+0000 7f43aa387640 1 -- 172.21.15.23:0/77184658 wait complete. 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 Processor -- start 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 -- start start 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a4069260 0x7f43a4198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a41076f0 0x7f43a419c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4111a00 con 0x7f43a40688a0 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4111880 con 0x7f43a4069260 2024-03-28T16:03:48.384 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43aa387640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a4111b80 con 0x7f43a41076f0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43a3fff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:59692/0 (socket says 172.21.15.23:59692) 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43a3fff640 1 -- 172.21.15.23:0/3336618927 learned_addr learned my addr 172.21.15.23:0/3336618927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43a8924640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a41076f0 0x7f43a419c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.381+0000 7f43a37fe640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a4069260 0x7f43a4198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 -- 172.21.15.23:0/3336618927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a41076f0 msgr2=0x7f43a419c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a41076f0 0x7f43a419c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 -- 172.21.15.23:0/3336618927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a4069260 msgr2=0x7f43a4198480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a4069260 0x7f43a4198480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 -- 172.21.15.23:0/3336618927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a419cf30 con 0x7f43a40688a0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a3fff640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f439800ea60 tx=0x7f439800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439800cde0 con 0x7f43a40688a0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43980035f0 con 0x7f43a40688a0 2024-03-28T16:03:48.385 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a419d220 con 0x7f43a40688a0 2024-03-28T16:03:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4398010670 con 0x7f43a40688a0 2024-03-28T16:03:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.382+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a41a4ab0 con 0x7f43a40688a0 2024-03-28T16:03:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.383+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a4068490 con 0x7f43a40688a0 2024-03-28T16:03:48.386 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.384+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f43980037a0 con 0x7f43a40688a0 2024-03-28T16:03:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.385+0000 7f43a17fa640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.385+0000 7f43a37fe640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.385+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f43980cf630 con 0x7f43a40688a0 2024-03-28T16:03:48.387 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.385+0000 7f43a37fe640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 0x7f438c0a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4394009230 tx=0x7f4394000bc0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:48.388 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.388+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f439809c000 con 0x7f43a40688a0 2024-03-28T16:03:48.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.602+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f43a4199290 con 0x7f43a40688a0 2024-03-28T16:03:48.602 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.602+0000 7f43a17fa640 1 -- 172.21.15.23:0/3336618927 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) v1 ==== 81+0+944 (secure 0 0 0) 0x7f439809b900 con 0x7f43a40688a0 2024-03-28T16:03:48.602 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1859625110 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3007922889 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/1340727098 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2505463471 2024-03-29T15:43:34.285406+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/3395482026 2024-03-29T15:43:34.285406+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/4208679988 2024-03-29T15:43:34.285406+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/4155829879 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6800/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/1340727098 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/2095255310 2024-03-29T15:43:06.191436+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/421542303 2024-03-29T15:53:05.125385+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:6801/3484701507 2024-03-29T15:43:06.191436+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1534223304 2024-03-29T15:43:06.191436+0000 2024-03-28T16:03:48.603 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/611179097 2024-03-29T15:43:06.191436+0000 2024-03-28T16:03:48.604 INFO:teuthology.orchestra.run.smithi023.stdout:172.21.15.23:0/1777550602 2024-03-29T15:43:34.285406+0000 2024-03-28T16:03:48.604 INFO:teuthology.orchestra.run.smithi023.stderr:listed 16 entries 2024-03-28T16:03:48.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 msgr2=0x7f438c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.604 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 0x7f438c0a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4394009230 tx=0x7f4394000bc0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 msgr2=0x7f43a4197f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f439800ea60 tx=0x7f439800ef20 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 shutdown_connections 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f438c0a2970 0x7f438c0a4e30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f43a41076f0 0x7f43a419c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a4069260 0x7f43a4198480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 --2- 172.21.15.23:0/3336618927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f43a40688a0 0x7f43a4197f40 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 >> 172.21.15.23:0/3336618927 conn(0x7f43a40776a0 msgr2=0x7f43a40fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 shutdown_connections 2024-03-28T16:03:48.605 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:48.604+0000 7f43aa387640 1 -- 172.21.15.23:0/3336618927 wait complete. 2024-03-28T16:03:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:48 smithi023 ceph-mon[32448]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/4209309522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/673074223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.614 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:48 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1155633198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.615 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-28T16:03:48.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:48.615 DEBUG:teuthology.orchestra.run.smithi023:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-28T16:03:48.655 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T16:03:48.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:48.656 DEBUG:teuthology.orchestra.run.smithi023:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-28T16:03:48.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:48.710 DEBUG:teuthology.orchestra.run.smithi023:> ip netns list 2024-03-28T16:03:48.764 INFO:teuthology.orchestra.run.smithi023.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-28T16:03:48.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:48.765 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-28T16:03:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:48 smithi117 ceph-mon[36002]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/4209309522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/673074223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:48 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1155633198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-28T16:03:48.832 DEBUG:teuthology.orchestra.run.smithi023:> sudo ip link delete ceph-brx 2024-03-28T16:03:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:48 smithi120 ceph-mon[36826]: pgmap v332: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 12 op/s 2024-03-28T16:03:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/4209309522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/673074223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:48 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1155633198' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-28T16:03:48.921 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-28T16:03:48.932 INFO:tasks.cephadm:Teardown begin 2024-03-28T16:03:48.933 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-28T16:03:48.933 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- 172.21.15.23:0/3918416248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 msgr2=0x7f147010e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 --2- 172.21.15.23:0/3918416248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147010e180 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1468009ca0 tx=0x7f146802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- 172.21.15.23:0/3918416248 shutdown_connections 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 --2- 172.21.15.23:0/3918416248 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147010e180 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 --2- 172.21.15.23:0/3918416248 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470103230 0x7f147010b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 --2- 172.21.15.23:0/3918416248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470102910 0x7f1470102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.043 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- 172.21.15.23:0/3918416248 >> 172.21.15.23:0/3918416248 conn(0x7f14700776a0 msgr2=0x7f1470077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- 172.21.15.23:0/3918416248 shutdown_connections 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- 172.21.15.23:0/3918416248 wait complete. 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 Processor -- start 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.040+0000 7f1475f0c640 1 -- start start 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470102910 0x7f1470197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470111ae0 con 0x7f147010bce0 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470111960 con 0x7f1470103230 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470111c60 con 0x7f1470102910 2024-03-28T16:03:49.044 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.23:41568/0 (socket says 172.21.15.23:41568) 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146f7fe640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470102910 0x7f1470197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54568/0 (socket says 172.21.15.23:54568) 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 -- 172.21.15.23:0/1067362429 learned_addr learned my addr 172.21.15.23:0/1067362429 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 -- 172.21.15.23:0/1067362429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470102910 msgr2=0x7f1470197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470102910 0x7f1470197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 -- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 msgr2=0x7f147019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.045 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 -- 172.21.15.23:0/1067362429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f147019cf50 con 0x7f1470103230 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146ffff640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146effd640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f145800e9f0 tx=0x7f145800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800cd80 con 0x7f1470103230 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.041+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147019d240 con 0x7f1470103230 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.042+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1458005500 con 0x7f1470103230 2024-03-28T16:03:49.046 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.042+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458010690 con 0x7f1470103230 2024-03-28T16:03:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.042+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701a4ad0 con 0x7f1470103230 2024-03-28T16:03:49.047 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.042+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470068490 con 0x7f1470103230 2024-03-28T16:03:49.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.044+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f14580046c0 con 0x7f1470103230 2024-03-28T16:03:49.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.044+0000 7f146cff9640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 0x7f145c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.044+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f14580cf860 con 0x7f1470103230 2024-03-28T16:03:49.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.044+0000 7f146f7fe640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 0x7f145c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.051 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.045+0000 7f146f7fe640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 0x7f145c0a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1460009bd0 tx=0x7f1460005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.052 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.052+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1458053740 con 0x7f1470103230 2024-03-28T16:03:49.279 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.279+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14701a4d00 con 0x7f1470103230 2024-03-28T16:03:49.280 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.279+0000 7f146cff9640 1 -- 172.21.15.23:0/1067362429 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f145809c1a0 con 0x7f1470103230 2024-03-28T16:03:49.281 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:49.281 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:52:55.819433+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[14640,24454,24455,24481,24494]},"id":1}]} 2024-03-28T16:03:49.281 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-28T16:03:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 msgr2=0x7f145c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 0x7f145c0a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1460009bd0 tx=0x7f1460005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 msgr2=0x7f14701984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f145800e9f0 tx=0x7f145800edf0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.282 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 shutdown_connections 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f145c0a2970 0x7f145c0a4e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f147010bce0 0x7f147019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1470103230 0x7f14701984a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 --2- 172.21.15.23:0/1067362429 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f1470102910 0x7f1470197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 >> 172.21.15.23:0/1067362429 conn(0x7f14700776a0 msgr2=0x7f14700fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 shutdown_connections 2024-03-28T16:03:49.283 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.282+0000 7f1475f0c640 1 -- 172.21.15.23:0/1067362429 wait complete. 2024-03-28T16:03:49.294 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 -- 172.21.15.23:0/2986339499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144106730 msgr2=0x7f5144106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 --2- 172.21.15.23:0/2986339499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144106730 0x7f5144106b10 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f5140009900 tx=0x7f514002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 -- 172.21.15.23:0/2986339499 shutdown_connections 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 --2- 172.21.15.23:0/2986339499 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144100f60 0x7f514410d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 --2- 172.21.15.23:0/2986339499 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f5144100a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 --2- 172.21.15.23:0/2986339499 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144106730 0x7f5144106b10 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 -- 172.21.15.23:0/2986339499 >> 172.21.15.23:0/2986339499 conn(0x7f51440776a0 msgr2=0x7f5144077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 -- 172.21.15.23:0/2986339499 shutdown_connections 2024-03-28T16:03:49.400 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.397+0000 7f514c56f640 1 -- 172.21.15.23:0/2986339499 wait complete. 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 Processor -- start 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 -- start start 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f514419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144106730 0x7f514419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51441109a0 con 0x7f5144100f60 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144110820 con 0x7f5144106730 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514c56f640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5144110b20 con 0x7f51441005a0 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514a30b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f514419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54584/0 (socket says 172.21.15.23:54584) 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 -- 172.21.15.23:0/3896476932 learned_addr learned my addr 172.21.15.23:0/3896476932 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514ab0c640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144106730 0x7f514419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.401 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 -- 172.21.15.23:0/3896476932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 msgr2=0x7f514419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f514419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 -- 172.21.15.23:0/3896476932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144106730 msgr2=0x7f514419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144106730 0x7f514419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 -- 172.21.15.23:0/3896476932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514419f100 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f514a30b640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f514419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.398+0000 7f5149b0a640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f513400e9f0 tx=0x7f513400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.399+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513400cd80 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.399+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5134005500 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.399+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514419f3f0 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.399+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5134010690 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.399+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51441a6c80 con 0x7f5144100f60 2024-03-28T16:03:49.402 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.400+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5144068490 con 0x7f5144100f60 2024-03-28T16:03:49.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.402+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f51340046c0 con 0x7f5144100f60 2024-03-28T16:03:49.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.402+0000 7f513b7fe640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 0x7f51300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.403 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.402+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f51340cf8c0 con 0x7f5144100f60 2024-03-28T16:03:49.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.402+0000 7f514a30b640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 0x7f51300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.404 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.403+0000 7f514a30b640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 0x7f51300a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5140009230 tx=0x7f5140005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.405 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.405+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5134053740 con 0x7f5144100f60 2024-03-28T16:03:49.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.628+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514419b460 con 0x7f5144100f60 2024-03-28T16:03:49.629 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.629+0000 7f513b7fe640 1 -- 172.21.15.23:0/3896476932 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11459 (secure 0 0 0) 0x7f514419b460 con 0x7f5144100f60 2024-03-28T16:03:49.630 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:49.630 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14553,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6835/1890623637","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6834","nonce":1890623637},{"type":"v1","addr":"172.21.15.23:6835","nonce":1890623637}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.23:6837/252463346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6836","nonce":252463346},{"type":"v1","addr":"172.21.15.23:6837","nonce":252463346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-28T15:50:31.785133+0000","modified":"2024-03-28T15:52:55.819433+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24494,"mds_1":24481,"mds_2":24455,"mds_3":24454,"mds_4":14640},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14628":{"gid":14628,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.23:6839/1641251810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6838","nonce":1641251810},{"type":"v1","addr":"172.21.15.23:6839","nonce":1641251810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14640":{"gid":14640,"name":"a","rank":4,"incarnation":26,"state":"up:active","state_seq":10,"addr":"172.21.15.23:6841/910694674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6840","nonce":910694674},{"type":"v1","addr":"172.21.15.23:6841","nonce":910694674}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24392":{"gid":24392,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6833/2721990059","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":2721990059},{"type":"v1","addr":"172.21.15.117:6833","nonce":2721990059}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24403":{"gid":24403,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6833/515874083","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6832","nonce":515874083},{"type":"v1","addr":"172.21.15.120:6833","nonce":515874083}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24422":{"gid":24422,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6835/3656482331","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3656482331},{"type":"v1","addr":"172.21.15.117:6835","nonce":3656482331}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24442":{"gid":24442,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.120:6835/1615812668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6834","nonce":1615812668},{"type":"v1","addr":"172.21.15.120:6835","nonce":1615812668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24454":{"gid":24454,"name":"c","rank":3,"incarnation":24,"state":"up:active","state_seq":13,"addr":"172.21.15.120:6837/2763171574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6836","nonce":2763171574},{"type":"v1","addr":"172.21.15.120:6837","nonce":2763171574}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24455":{"gid":24455,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":12,"addr":"172.21.15.117:6837/30575642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":30575642},{"type":"v1","addr":"172.21.15.117:6837","nonce":30575642}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":11,"addr":"172.21.15.120:6839/4092312487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6838","nonce":4092312487},{"type":"v1","addr":"172.21.15.120:6839","nonce":4092312487}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24494":{"gid":24494,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6839/3971000510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3971000510},{"type":"v1","addr":"172.21.15.117:6839","nonce":3971000510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24494,"qdb_cluster":[14640,24454,24455,24481,24494]},"id":1}]} 2024-03-28T16:03:49.630 INFO:teuthology.orchestra.run.smithi023.stderr:dumped fsmap epoch 29 2024-03-28T16:03:49.631 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 msgr2=0x7f51300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 0x7f51300a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5140009230 tx=0x7f5140005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 msgr2=0x7f514419a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f513400e9f0 tx=0x7f513400edf0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 shutdown_connections 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f51300a2970 0x7f51300a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5144106730 0x7f514419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5144100f60 0x7f514419a650 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 --2- 172.21.15.23:0/3896476932 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51441005a0 0x7f514419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 >> 172.21.15.23:0/3896476932 conn(0x7f51440776a0 msgr2=0x7f5144068800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 shutdown_connections 2024-03-28T16:03:49.632 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.631+0000 7f514c56f640 1 -- 172.21.15.23:0/3896476932 wait complete. 2024-03-28T16:03:49.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:49 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3336618927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:03:49.641 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:49 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1067362429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:49.642 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 -- 172.21.15.23:0/3737767664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51500691c0 msgr2=0x7f515010e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 --2- 172.21.15.23:0/3737767664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51500691c0 0x7f515010e310 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5148009cd0 tx=0x7f514802c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 -- 172.21.15.23:0/3737767664 shutdown_connections 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 --2- 172.21.15.23:0/3737767664 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f51500691c0 0x7f515010e310 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 --2- 172.21.15.23:0/3737767664 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f5150068800 0x7f5150068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.749 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 --2- 172.21.15.23:0/3737767664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150107590 0x7f5150107970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 -- 172.21.15.23:0/3737767664 >> 172.21.15.23:0/3737767664 conn(0x7f5150075fb0 msgr2=0x7f51500763c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.746+0000 7f5154d9d640 1 -- 172.21.15.23:0/3737767664 shutdown_connections 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 -- 172.21.15.23:0/3737767664 wait complete. 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 Processor -- start 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 -- start start 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150068800 0x7f5150199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51500691c0 0x7f515019a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5150111880 con 0x7f5150107590 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5150111700 con 0x7f5150068800 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.747+0000 7f5154d9d640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5150111a00 con 0x7f51500691c0 2024-03-28T16:03:49.750 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514f7fe640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150068800 0x7f5150199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54594/0 (socket says 172.21.15.23:54594) 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 -- 172.21.15.23:0/1578911927 learned_addr learned my addr 172.21.15.23:0/1578911927 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514effd640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51500691c0 0x7f515019a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 -- 172.21.15.23:0/1578911927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51500691c0 msgr2=0x7f515019a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51500691c0 0x7f515019a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 -- 172.21.15.23:0/1578911927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150068800 msgr2=0x7f5150199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150068800 0x7f5150199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 -- 172.21.15.23:0/1578911927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515019eee0 con 0x7f5150107590 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514ffff640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f51480348f0 tx=0x7f5148032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148041020 con 0x7f5150107590 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5148032e80 con 0x7f5150107590 2024-03-28T16:03:49.751 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515019f170 con 0x7f5150107590 2024-03-28T16:03:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148039040 con 0x7f5150107590 2024-03-28T16:03:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.748+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51501a6a60 con 0x7f5150107590 2024-03-28T16:03:49.752 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.749+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51500683b0 con 0x7f5150107590 2024-03-28T16:03:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.750+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5148044020 con 0x7f5150107590 2024-03-28T16:03:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.751+0000 7f514cff9640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 0x7f513c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.751+0000 7f514f7fe640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 0x7f513c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.751+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f51480f0c80 con 0x7f5150107590 2024-03-28T16:03:49.753 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.752+0000 7f514f7fe640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 0x7f513c0a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5140009bd0 tx=0x7f5140005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:49.756 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.755+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f514804b020 con 0x7f5150107590 2024-03-28T16:03:49.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:49 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3336618927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:03:49.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:49 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1067362429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:49 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3336618927' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-28T16:03:49.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:49 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1067362429' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:49.940 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.940+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51501a6c90 con 0x7f5150107590 2024-03-28T16:03:49.942 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.942+0000 7f514cff9640 1 -- 172.21.15.23:0/1578911927 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25476 (secure 0 0 0) 0x7f51480bd6d0 con 0x7f5150107590 2024-03-28T16:03:49.942 INFO:teuthology.orchestra.run.smithi023.stdout: 2024-03-28T16:03:49.943 INFO:teuthology.orchestra.run.smithi023.stdout:{"epoch":103,"fsid":"a96af96e-ed19-11ee-95d0-87774f69a715","created":"2024-03-28T15:42:38.302272+0000","modified":"2024-03-28T16:03:14.372495+0000","last_up_change":"2024-03-28T15:49:44.285478+0000","last_in_change":"2024-03-28T15:49:24.477049+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T15:45:58.632126+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-28T15:50:30.115107+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-28T15:50:30.959648+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-28T15:50:34.679185+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d73a4797-07a2-4f5c-9df2-ae9b5ec8c617","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6802","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6803","nonce":3911626101}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6804","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6805","nonce":3911626101}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6808","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6809","nonce":3911626101}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6806","nonce":3911626101},{"type":"v1","addr":"172.21.15.23:6807","nonce":3911626101}]},"public_addr":"172.21.15.23:6803/3911626101","cluster_addr":"172.21.15.23:6805/3911626101","heartbeat_back_addr":"172.21.15.23:6809/3911626101","heartbeat_front_addr":"172.21.15.23:6807/3911626101","state":["exists","up"]},{"osd":1,"uuid":"6fe2a5d7-6406-4527-bf36-1ad15e345a54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6801","nonce":1603075962}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6803","nonce":1603075962}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6807","nonce":1603075962}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1603075962},{"type":"v1","addr":"172.21.15.117:6805","nonce":1603075962}]},"public_addr":"172.21.15.117:6801/1603075962","cluster_addr":"172.21.15.117:6803/1603075962","heartbeat_back_addr":"172.21.15.117:6807/1603075962","heartbeat_front_addr":"172.21.15.117:6805/1603075962","state":["exists","up"]},{"osd":2,"uuid":"9c0639e5-8825-479c-a148-f3a525d6995c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6800","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6801","nonce":399817825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6802","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6803","nonce":399817825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6806","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6807","nonce":399817825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6804","nonce":399817825},{"type":"v1","addr":"172.21.15.120:6805","nonce":399817825}]},"public_addr":"172.21.15.120:6801/399817825","cluster_addr":"172.21.15.120:6803/399817825","heartbeat_back_addr":"172.21.15.120:6807/399817825","heartbeat_front_addr":"172.21.15.120:6805/399817825","state":["exists","up"]},{"osd":3,"uuid":"a5757ad6-b03a-4985-ba33-e0010a2c3161","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6810","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6811","nonce":4165771629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6812","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6813","nonce":4165771629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6816","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6817","nonce":4165771629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6814","nonce":4165771629},{"type":"v1","addr":"172.21.15.23:6815","nonce":4165771629}]},"public_addr":"172.21.15.23:6811/4165771629","cluster_addr":"172.21.15.23:6813/4165771629","heartbeat_back_addr":"172.21.15.23:6817/4165771629","heartbeat_front_addr":"172.21.15.23:6815/4165771629","state":["exists","up"]},{"osd":4,"uuid":"884d0555-eb5d-4206-8931-26bf05fe7e40","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6809","nonce":1702661707}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6811","nonce":1702661707}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6815","nonce":1702661707}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1702661707},{"type":"v1","addr":"172.21.15.117:6813","nonce":1702661707}]},"public_addr":"172.21.15.117:6809/1702661707","cluster_addr":"172.21.15.117:6811/1702661707","heartbeat_back_addr":"172.21.15.117:6815/1702661707","heartbeat_front_addr":"172.21.15.117:6813/1702661707","state":["exists","up"]},{"osd":5,"uuid":"0743e053-1af5-44cd-9d42-efd7079b15f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6808","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6809","nonce":3042970364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6810","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6811","nonce":3042970364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6814","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6815","nonce":3042970364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6812","nonce":3042970364},{"type":"v1","addr":"172.21.15.120:6813","nonce":3042970364}]},"public_addr":"172.21.15.120:6809/3042970364","cluster_addr":"172.21.15.120:6811/3042970364","heartbeat_back_addr":"172.21.15.120:6815/3042970364","heartbeat_front_addr":"172.21.15.120:6813/3042970364","state":["exists","up"]},{"osd":6,"uuid":"21218a07-2898-42e9-a82b-935c3bb50fcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6818","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6819","nonce":2799436547}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6820","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6821","nonce":2799436547}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6824","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6825","nonce":2799436547}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6822","nonce":2799436547},{"type":"v1","addr":"172.21.15.23:6823","nonce":2799436547}]},"public_addr":"172.21.15.23:6819/2799436547","cluster_addr":"172.21.15.23:6821/2799436547","heartbeat_back_addr":"172.21.15.23:6825/2799436547","heartbeat_front_addr":"172.21.15.23:6823/2799436547","state":["exists","up"]},{"osd":7,"uuid":"18720d2e-c5ab-4b3e-b8ee-8b2b687daabf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6817","nonce":3265640789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6819","nonce":3265640789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6823","nonce":3265640789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":3265640789},{"type":"v1","addr":"172.21.15.117:6821","nonce":3265640789}]},"public_addr":"172.21.15.117:6817/3265640789","cluster_addr":"172.21.15.117:6819/3265640789","heartbeat_back_addr":"172.21.15.117:6823/3265640789","heartbeat_front_addr":"172.21.15.117:6821/3265640789","state":["exists","up"]},{"osd":8,"uuid":"692e7ca1-8a89-4b1d-a931-f7e9bb788110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6816","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6817","nonce":2425649458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6818","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6819","nonce":2425649458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6822","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6823","nonce":2425649458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6820","nonce":2425649458},{"type":"v1","addr":"172.21.15.120:6821","nonce":2425649458}]},"public_addr":"172.21.15.120:6817/2425649458","cluster_addr":"172.21.15.120:6819/2425649458","heartbeat_back_addr":"172.21.15.120:6823/2425649458","heartbeat_front_addr":"172.21.15.120:6821/2425649458","state":["exists","up"]},{"osd":9,"uuid":"30e08565-5a2b-4c5f-a1de-bd6ef74ae4a4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6826","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6827","nonce":3759918369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6828","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6829","nonce":3759918369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6832","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6833","nonce":3759918369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.23:6830","nonce":3759918369},{"type":"v1","addr":"172.21.15.23:6831","nonce":3759918369}]},"public_addr":"172.21.15.23:6827/3759918369","cluster_addr":"172.21.15.23:6829/3759918369","heartbeat_back_addr":"172.21.15.23:6833/3759918369","heartbeat_front_addr":"172.21.15.23:6831/3759918369","state":["exists","up"]},{"osd":10,"uuid":"52537bc3-64df-47b7-85c8-6b32eddd8296","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6825","nonce":2718715986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6827","nonce":2718715986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6831","nonce":2718715986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2718715986},{"type":"v1","addr":"172.21.15.117:6829","nonce":2718715986}]},"public_addr":"172.21.15.117:6825/2718715986","cluster_addr":"172.21.15.117:6827/2718715986","heartbeat_back_addr":"172.21.15.117:6831/2718715986","heartbeat_front_addr":"172.21.15.117:6829/2718715986","state":["exists","up"]},{"osd":11,"uuid":"d7c32944-dd01-42b8-9b33-5aed77590035","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6824","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6825","nonce":3052698100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6826","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6827","nonce":3052698100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6830","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6831","nonce":3052698100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.120:6828","nonce":3052698100},{"type":"v1","addr":"172.21.15.120:6829","nonce":3052698100}]},"public_addr":"172.21.15.120:6825/3052698100","cluster_addr":"172.21.15.120:6827/3052698100","heartbeat_back_addr":"172.21.15.120:6831/3052698100","heartbeat_front_addr":"172.21.15.120:6829/3052698100","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:08.891906+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:32.211797+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:45:55.679133+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:20.883525+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:46:45.719136+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:09.707223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:34.821774+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:47:59.966499+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:25.335490+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:48:51.578432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:16.798089+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-28T15:49:42.623338+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.0","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.1d","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.23:0/1859625110":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/3007922889":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2505463471":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/3395482026":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6801/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:6800/4208679988":"2024-03-29T15:43:34.285406+0000","172.21.15.23:0/4155829879":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6800/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:6801/1340727098":"2024-03-29T15:53:05.125385+0000","172.21.15.23:0/2095255310":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/421542303":"2024-03-29T15:53:05.125385+0000","172.21.15.23:6801/3484701507":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1534223304":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/611179097":"2024-03-29T15:43:06.191436+0000","172.21.15.23:0/1777550602":"2024-03-29T15:43:34.285406+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 msgr2=0x7f513c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 0x7f513c0a4eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5140009bd0 tx=0x7f5140005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 msgr2=0x7f515019e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f51480348f0 tx=0x7f5148032c90 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 shutdown_connections 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f513c0a29f0 0x7f513c0a4eb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f5150107590 0x7f515019e7b0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f51500691c0 0x7f515019a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 --2- 172.21.15.23:0/1578911927 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5150068800 0x7f5150199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 >> 172.21.15.23:0/1578911927 conn(0x7f5150075fb0 msgr2=0x7f515010c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 shutdown_connections 2024-03-28T16:03:49.945 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:49.944+0000 7f5154d9d640 1 -- 172.21.15.23:0/1578911927 wait complete. 2024-03-28T16:03:49.958 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-28T16:03:50.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 -- 172.21.15.23:0/1553767593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 msgr2=0x7f23f8101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 --2- 172.21.15.23:0/1553767593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8101530 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f23e00098c0 tx=0x7f23e002bb50 comp rx=0 tx=0).stop 2024-03-28T16:03:50.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 -- 172.21.15.23:0/1553767593 shutdown_connections 2024-03-28T16:03:50.065 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 --2- 172.21.15.23:0/1553767593 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8101a70 0x7f23f810d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 --2- 172.21.15.23:0/1553767593 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8101530 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 --2- 172.21.15.23:0/1553767593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8105540 0x7f23f8105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 -- 172.21.15.23:0/1553767593 >> 172.21.15.23:0/1553767593 conn(0x7f23f8075800 msgr2=0x7f23f8075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 -- 172.21.15.23:0/1553767593 shutdown_connections 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 -- 172.21.15.23:0/1553767593 wait complete. 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.063+0000 7f23fd13b640 1 Processor -- start 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 -- start start 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8198d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 0x7f23f81992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8110970 con 0x7f23f8105540 2024-03-28T16:03:50.066 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f81107f0 con 0x7f23f8101a70 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23fd13b640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f8110af0 con 0x7f23f81010b0 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54610/0 (socket says 172.21.15.23:54610) 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 -- 172.21.15.23:0/239781922 learned_addr learned my addr 172.21.15.23:0/239781922 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23effff640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 0x7f23f81992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f6d9d640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8198d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 -- 172.21.15.23:0/239781922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 msgr2=0x7f23f8198d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8198d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 -- 172.21.15.23:0/239781922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 msgr2=0x7f23f81992b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 0x7f23f81992b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 -- 172.21.15.23:0/239781922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f81a4330 con 0x7f23f8105540 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23effff640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 0x7f23f81992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.064+0000 7f23f759e640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f23f000efc0 tx=0x7f23f000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:50.067 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.065+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f0003970 con 0x7f23f8105540 2024-03-28T16:03:50.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.065+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f81a4560 con 0x7f23f8105540 2024-03-28T16:03:50.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.065+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23f0004020 con 0x7f23f8105540 2024-03-28T16:03:50.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.065+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f000b9f0 con 0x7f23f8105540 2024-03-28T16:03:50.068 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.065+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f81a4a30 con 0x7f23f8105540 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.066+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f80688e0 con 0x7f23f8105540 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.067+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f23f001d020 con 0x7f23f8105540 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.067+0000 7f23f4d99640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 0x7f23e40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.068+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9583+0+0 (secure 0 0 0) 0x7f23f00cf700 con 0x7f23f8105540 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.068+0000 7f23f6d9d640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 0x7f23e40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.069 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.068+0000 7f23f6d9d640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 0x7f23e40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23f8076e20 tx=0x7f23e8005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:50.072 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.072+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f23f009c040 con 0x7f23f8105540 2024-03-28T16:03:50.296 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.296+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f23f802d070 con 0x7f23f8105540 2024-03-28T16:03:50.302 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.302+0000 7f23f4d99640 1 -- 172.21.15.23:0/239781922 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7f23f009b940 con 0x7f23f8105540 2024-03-28T16:03:50.302 INFO:teuthology.orchestra.run.smithi023.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-28T16:03:50.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 msgr2=0x7f23e40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 0x7f23e40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f23f8076e20 tx=0x7f23e8005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 msgr2=0x7f23f81a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f23f000efc0 tx=0x7f23f000c800 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 shutdown_connections 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f23e40a2970 0x7f23e40a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f23f8105540 0x7f23f81a3df0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23f8101a70 0x7f23f81992b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 --2- 172.21.15.23:0/239781922 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f23f81010b0 0x7f23f8198d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 >> 172.21.15.23:0/239781922 conn(0x7f23f8075800 msgr2=0x7f23f80fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 shutdown_connections 2024-03-28T16:03:50.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.306+0000 7f23fd13b640 1 -- 172.21.15.23:0/239781922 wait complete. 2024-03-28T16:03:50.319 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 -- 172.21.15.23:0/2356639996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34341065f0 msgr2=0x7f34341069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 --2- 172.21.15.23:0/2356639996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34341065f0 0x7f34341069d0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3424009900 tx=0x7f342402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 -- 172.21.15.23:0/2356639996 shutdown_connections 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 --2- 172.21.15.23:0/2356639996 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f3434100ef0 0x7f343410d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 --2- 172.21.15.23:0/2356639996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 0x7f34341009b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 --2- 172.21.15.23:0/2356639996 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f34341065f0 0x7f34341069d0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 -- 172.21.15.23:0/2356639996 >> 172.21.15.23:0/2356639996 conn(0x7f34340776a0 msgr2=0x7f3434077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 -- 172.21.15.23:0/2356639996 shutdown_connections 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.432+0000 7f343a937640 1 -- 172.21.15.23:0/2356639996 wait complete. 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 Processor -- start 2024-03-28T16:03:50.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 -- start start 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 0x7f3434198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 0x7f34341a3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34341108d0 con 0x7f3434100ef0 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3434110750 con 0x7f3434100530 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f343a937640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3434110a50 con 0x7f34341065f0 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54674/0 (socket says 172.21.15.23:54674) 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 -- 172.21.15.23:0/1974028380 learned_addr learned my addr 172.21.15.23:0/1974028380 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:50.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f3438ed4640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 0x7f34341a3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f3433fff640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 0x7f3434198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 -- 172.21.15.23:0/1974028380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 msgr2=0x7f34341a3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 0x7f34341a3cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 -- 172.21.15.23:0/1974028380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 msgr2=0x7f3434198c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 0x7f3434198c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f34337fe640 1 -- 172.21.15.23:0/1974028380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34341a4230 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.433+0000 7f3438ed4640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 0x7f34341a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f34337fe640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f342800c920 tx=0x7f342800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428007c70 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34341a4400 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3428007e10 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428018890 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.434+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34341a4970 con 0x7f3434100ef0 2024-03-28T16:03:50.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.435+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3434068490 con 0x7f3434100ef0 2024-03-28T16:03:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.436+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3428018a70 con 0x7f3434100ef0 2024-03-28T16:03:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.436+0000 7f34317fa640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 0x7f341c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.436+0000 7f3433fff640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 0x7f341c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.436+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9925+0+0 (secure 0 0 0) 0x7f34280cf490 con 0x7f3434100ef0 2024-03-28T16:03:50.438 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.437+0000 7f3433fff640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 0x7f341c0a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3424009230 tx=0x7f3424005fd0 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:50.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.441+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3428014030 con 0x7f3434100ef0 2024-03-28T16:03:50.597 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.597 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.597 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mds[73552]: -- 172.21.15.23:0/343191279 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557d0819fa40 con 0x557d08172800 2024-03-28T16:03:50.597 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: starting mds.a at 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a[32444]: 2024-03-28T16:03:50.297+0000 7f5719360700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/3896476932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1578911927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mon[32448]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T16:03:50.598 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.599 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.599 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-mds[73190]: -- 172.21.15.23:0/663324418 <== mon.0 v2:172.21.15.23:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e2ed71fa40 con 0x55e2ed6f2c00 2024-03-28T16:03:50.599 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:03:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: starting mds.g at 2024-03-28T16:03:50.668 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:50.668+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f34341089e0 con 0x7f3434100ef0 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/3896476932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1578911927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T16:03:50.723 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mon[36002]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mds[66327]: -- 172.21.15.117:0/3002176444 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559f67db3a40 con 0x559f67d86800 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: starting mds.b at 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.724 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mds[65701]: -- 172.21.15.117:0/1552239255 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5574203b9a40 con 0x55742038d000 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: starting mds.e at 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mds[66012]: -- 172.21.15.117:0/2539719462 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56000c5fba40 con 0x56000c5ce800 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: starting mds.h at 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-mds[65380]: -- 172.21.15.117:0/3477045767 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560cfe291a40 con 0x560cfe264c00 2024-03-28T16:03:50.725 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:03:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: starting mds.k at 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mds[66179]: -- 172.21.15.120:0/4064739888 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560939cb9a40 con 0x560939c8c800 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: starting mds.l at 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i[67098]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i[67098]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mds[67102]: -- 172.21.15.120:0/2868407697 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e358f3fa40 con 0x55e358f13000 2024-03-28T16:03:50.740 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i[67098]: starting mds.i at 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: pgmap v333: 129 pgs: 129 active+clean; 2.3 MiB data, 506 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/3896476932' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1578911927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/239781922' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: osdmap e104: 12 total, 12 up, 12 in 2024-03-28T16:03:50.741 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mon[36826]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mds[66788]: -- 172.21.15.120:0/4062622487 <== mon.1 v2:172.21.15.120:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f9a39ada40 con 0x55f9a3980800 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: starting mds.c at 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: ignoring --setuser ceph since I am not root 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: ignoring --setgroup ceph since I am not root 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-mds[66483]: -- 172.21.15.120:0/2500419483 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563e7b1eda40 con 0x563e7b1c0800 2024-03-28T16:03:50.742 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:03:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: starting mds.f at 2024-03-28T16:03:51.304 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.303+0000 7f34317fa640 1 -- 172.21.15.23:0/1974028380 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f342809bc70 con 0x7f3434100ef0 2024-03-28T16:03:51.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 msgr2=0x7f341c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:51.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 0x7f341c0a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3424009230 tx=0x7f3424005fd0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.306 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 msgr2=0x7f34341991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f342800c920 tx=0x7f342800cde0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 shutdown_connections 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f341c0a2970 0x7f341c0a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f34341065f0 0x7f34341a3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f3434100ef0 0x7f34341991b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 --2- 172.21.15.23:0/1974028380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3434100530 0x7f3434198c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 >> 172.21.15.23:0/1974028380 conn(0x7f34340776a0 msgr2=0x7f34340fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 shutdown_connections 2024-03-28T16:03:51.307 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.306+0000 7f343a937640 1 -- 172.21.15.23:0/1974028380 wait complete. 2024-03-28T16:03:51.321 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 -- 172.21.15.23:0/1703360482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe406a630 msgr2=0x7f9fe410e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/1703360482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe406a630 0x7f9fe410e500 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc009cd0 tx=0x7f9fdc02c0f0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 -- 172.21.15.23:0/1703360482 shutdown_connections 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/1703360482 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe406a630 0x7f9fe410e500 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/1703360482 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4069c70 0x7f9fe406a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/1703360482 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe4107640 0x7f9fe4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.433 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 -- 172.21.15.23:0/1703360482 >> 172.21.15.23:0/1703360482 conn(0x7f9fe4077580 msgr2=0x7f9fe4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 -- 172.21.15.23:0/1703360482 shutdown_connections 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 -- 172.21.15.23:0/1703360482 wait complete. 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.430+0000 7f9feb2b2640 1 Processor -- start 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 -- start start 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe406a630 0x7f9fe4198410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 0x7f9fe419c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4111a10 con 0x7f9fe4069c70 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4111890 con 0x7f9fe4107640 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4111b90 con 0x7f9fe406a630 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54696/0 (socket says 172.21.15.23:54696) 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 -- 172.21.15.23:0/2677195979 learned_addr learned my addr 172.21.15.23:0/2677195979 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:51.434 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe984f640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 0x7f9fe419c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe884d640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe406a630 0x7f9fe4198410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 -- 172.21.15.23:0/2677195979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe406a630 msgr2=0x7f9fe4198410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe406a630 0x7f9fe4198410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 -- 172.21.15.23:0/2677195979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 msgr2=0x7f9fe419c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 0x7f9fe419c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 -- 172.21.15.23:0/2677195979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe419cec0 con 0x7f9fe4069c70 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe984f640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 0x7f9fe419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fe904e640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400d950 tx=0x7f9fd400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd4017030 con 0x7f9fe4069c70 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fd4004020 con 0x7f9fe4069c70 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.431+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe419d1b0 con 0x7f9fe4069c70 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.432+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40047a0 con 0x7f9fe4069c70 2024-03-28T16:03:51.435 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.432+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe41a4a40 con 0x7f9fe4069c70 2024-03-28T16:03:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.433+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4102db0 con 0x7f9fe4069c70 2024-03-28T16:03:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.435+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9fd40041d0 con 0x7f9fe4069c70 2024-03-28T16:03:51.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.435+0000 7f9fda7fc640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 0x7f9fd00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:51.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.435+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9925+0+0 (secure 0 0 0) 0x7f9fd40cedd0 con 0x7f9fe4069c70 2024-03-28T16:03:51.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.435+0000 7f9fe884d640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 0x7f9fd00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:51.437 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.436+0000 7f9fe884d640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 0x7f9fd00a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4106c60 tx=0x7f9fe0005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:51.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.443+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9fd409b5b0 con 0x7f9fe4069c70 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: Cluster is now healthy 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:03:51.579 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6836/4106723947,v1:172.21.15.120:6839/4106723947] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6837/1591013894,v1:172.21.15.120:6838/1591013894] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6838/105422057,v1:172.21.15.23:6839/105422057] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6832/1378798204,v1:172.21.15.120:6833/1378798204] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.23:6840/3912444342,v1:172.21.15.23:6841/3912444342] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6836/2646854866,v1:172.21.15.117:6838/2646854866] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6837/3804641324,v1:172.21.15.117:6839/3804641324] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6832/1843176215,v1:172.21.15.117:6835/1843176215] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.117:6833/2381164422,v1:172.21.15.117:6834/2381164422] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: mds.? [v2:172.21.15.120:6834/858101676,v1:172.21.15.120:6835/858101676] up:boot 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: fsmap 12 up:standby 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:03:51.580 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:03:51.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:03:51.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:03:51.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:03:51.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:03:51.581 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:51 smithi023 ceph-mon[32448]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:03:51.640 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:51.640+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9fe40688e0 con 0x7f9fe4069c70 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: Cluster is now healthy 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6836/4106723947,v1:172.21.15.120:6839/4106723947] up:boot 2024-03-28T16:03:51.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6837/1591013894,v1:172.21.15.120:6838/1591013894] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6838/105422057,v1:172.21.15.23:6839/105422057] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6832/1378798204,v1:172.21.15.120:6833/1378798204] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.23:6840/3912444342,v1:172.21.15.23:6841/3912444342] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6836/2646854866,v1:172.21.15.117:6838/2646854866] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6837/3804641324,v1:172.21.15.117:6839/3804641324] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6832/1843176215,v1:172.21.15.117:6835/1843176215] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.117:6833/2381164422,v1:172.21.15.117:6834/2381164422] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: mds.? [v2:172.21.15.120:6834/858101676,v1:172.21.15.120:6835/858101676] up:boot 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: fsmap 12 up:standby 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:03:51.738 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:03:51.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:03:51.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:03:51.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:03:51.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:03:51.739 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:51 smithi117 ceph-mon[36002]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:03:51.755 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: Cluster is now healthy 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1974028380' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6836/4106723947,v1:172.21.15.120:6839/4106723947] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6837/1591013894,v1:172.21.15.120:6838/1591013894] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6838/105422057,v1:172.21.15.23:6839/105422057] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6832/1378798204,v1:172.21.15.120:6833/1378798204] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.23:6840/3912444342,v1:172.21.15.23:6841/3912444342] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6836/2646854866,v1:172.21.15.117:6838/2646854866] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6837/3804641324,v1:172.21.15.117:6839/3804641324] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6832/1843176215,v1:172.21.15.117:6835/1843176215] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.117:6833/2381164422,v1:172.21.15.117:6834/2381164422] up:boot 2024-03-28T16:03:51.756 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: mds.? [v2:172.21.15.120:6834/858101676,v1:172.21.15.120:6835/858101676] up:boot 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: fsmap 12 up:standby 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-28T16:03:51.757 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:51 smithi120 ceph-mon[36826]: from='mgr.15546 172.21.15.23:0/4052724934' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-28T16:03:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.436+0000 7f9fda7fc640 1 -- 172.21.15.23:0/2677195979 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9fd409aeb0 con 0x7f9fe4069c70 2024-03-28T16:03:52.436 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_metadata' removed 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 msgr2=0x7f9fd00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 0x7f9fd00a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4106c60 tx=0x7f9fe0005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 msgr2=0x7f9fe4197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f9fd400d950 tx=0x7f9fd400de10 comp rx=0 tx=0).stop 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 shutdown_connections 2024-03-28T16:03:52.440 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f9fd00a2970 0x7f9fd00a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fe4107640 0x7f9fe419c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f9fe406a630 0x7f9fe4198410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 --2- 172.21.15.23:0/2677195979 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f9fe4069c70 0x7f9fe4197ed0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 >> 172.21.15.23:0/2677195979 conn(0x7f9fe4077580 msgr2=0x7f9fe40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:52.441 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 shutdown_connections 2024-03-28T16:03:52.443 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.440+0000 7f9feb2b2640 1 -- 172.21.15.23:0/2677195979 wait complete. 2024-03-28T16:03:52.462 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-28T16:03:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 -- 172.21.15.23:0/3761998787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 msgr2=0x7faf04068c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 --2- 172.21.15.23:0/3761998787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf04068c60 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf00009900 tx=0x7faf0002bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 -- 172.21.15.23:0/3761998787 shutdown_connections 2024-03-28T16:03:52.577 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 --2- 172.21.15.23:0/3761998787 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 0x7faf04110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 --2- 172.21.15.23:0/3761998787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 0x7faf0410db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 --2- 172.21.15.23:0/3761998787 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf04068c60 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 -- 172.21.15.23:0/3761998787 >> 172.21.15.23:0/3761998787 conn(0x7faf040776a0 msgr2=0x7faf04077ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 -- 172.21.15.23:0/3761998787 shutdown_connections 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.574+0000 7faf0bf22640 1 -- 172.21.15.23:0/3761998787 wait complete. 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 Processor -- start 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 -- start start 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 0x7faf0419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 0x7faf0419e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04113c00 con 0x7faf04068880 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04113a80 con 0x7faf040691a0 2024-03-28T16:03:52.578 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf0bf22640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf04113d80 con 0x7faf0410e040 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf09cbe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf09cbe640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54712/0 (socket says 172.21.15.23:54712) 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf09cbe640 1 -- 172.21.15.23:0/33758195 learned_addr learned my addr 172.21.15.23:0/33758195 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.575+0000 7faf094bd640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 0x7faf0419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf0a4bf640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 0x7faf0419e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 -- 172.21.15.23:0/33758195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 msgr2=0x7faf0419e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 0x7faf0419e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 -- 172.21.15.23:0/33758195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 msgr2=0x7faf0419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 0x7faf0419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 -- 172.21.15.23:0/33758195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0419f0f0 con 0x7faf04068880 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf09cbe640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faf00006010 tx=0x7faf0002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00041070 con 0x7faf04068880 2024-03-28T16:03:52.579 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf00004450 con 0x7faf04068880 2024-03-28T16:03:52.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0419f380 con 0x7faf04068880 2024-03-28T16:03:52.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf00003e00 con 0x7faf04068880 2024-03-28T16:03:52.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.576+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf041a6c70 con 0x7faf04068880 2024-03-28T16:03:52.580 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.577+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf04068490 con 0x7faf04068880 2024-03-28T16:03:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.578+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7faf00044070 con 0x7faf04068880 2024-03-28T16:03:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.579+0000 7faefaffd640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 0x7faef00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.579+0000 7faf094bd640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 0x7faef00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.579+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9371+0+0 (secure 0 0 0) 0x7faf000f0b00 con 0x7faf04068880 2024-03-28T16:03:52.581 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.579+0000 7faf094bd640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 0x7faef00a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf040fffa0 tx=0x7faef4005f90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:52.583 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.583+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faf000bcfd0 con 0x7faf04068880 2024-03-28T16:03:52.770 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:52.770+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faf041091d0 con 0x7faf04068880 2024-03-28T16:03:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:52 smithi117 ceph-mon[36002]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T16:03:52.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:52 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:52.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:52 smithi120 ceph-mon[36826]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T16:03:52.840 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:52 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:52 smithi023 ceph-mon[32448]: pgmap v335: 129 pgs: 129 active+clean; 2.3 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-28T16:03:52.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:52 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.450+0000 7faefaffd640 1 -- 172.21.15.23:0/33758195 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 169+0+0 (secure 0 0 0) 0x7faf000c1070 con 0x7faf04068880 2024-03-28T16:03:53.451 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data' removed 2024-03-28T16:03:53.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 msgr2=0x7faef00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:53.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 0x7faef00a4e30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faf040fffa0 tx=0x7faef4005f90 comp rx=0 tx=0).stop 2024-03-28T16:03:53.453 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 msgr2=0x7faf0419a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7faf00006010 tx=0x7faf0002cb30 comp rx=0 tx=0).stop 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 shutdown_connections 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7faef00a2970 0x7faef00a4e30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7faf0410e040 0x7faf0419e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faf040691a0 0x7faf0419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 --2- 172.21.15.23:0/33758195 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7faf04068880 0x7faf0419a100 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 >> 172.21.15.23:0/33758195 conn(0x7faf040776a0 msgr2=0x7faf0410f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 shutdown_connections 2024-03-28T16:03:53.454 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.453+0000 7faf0bf22640 1 -- 172.21.15.23:0/33758195 wait complete. 2024-03-28T16:03:53.470 DEBUG:teuthology.orchestra.run.smithi023:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 -- 172.21.15.23:0/2821100411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44881065d0 msgr2=0x7f44881069b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 --2- 172.21.15.23:0/2821100411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44881065d0 0x7f44881069b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4484009900 tx=0x7f448402bc30 comp rx=0 tx=0).stop 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 -- 172.21.15.23:0/2821100411 shutdown_connections 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 --2- 172.21.15.23:0/2821100411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4488101190 0x7f448810d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 --2- 172.21.15.23:0/2821100411 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881007f0 0x7f4488100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.586 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 --2- 172.21.15.23:0/2821100411 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f44881065d0 0x7f44881069b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 -- 172.21.15.23:0/2821100411 >> 172.21.15.23:0/2821100411 conn(0x7f4488077340 msgr2=0x7f4488077750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 -- 172.21.15.23:0/2821100411 shutdown_connections 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.583+0000 7f448fe19640 1 -- 172.21.15.23:0/2821100411 wait complete. 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 Processor -- start 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 -- start start 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44881007f0 0x7f4488197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 --2- >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 0x7f448819c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 -- --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488110920 con 0x7f4488101190 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44881107a0 con 0x7f44881007f0 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448fe19640 1 -- --> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488110aa0 con 0x7f44881065d0 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448d3b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448d3b4640 1 --2- >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.23:3300/0 says I am v2:172.21.15.23:54740/0 (socket says 172.21.15.23:54740) 2024-03-28T16:03:53.587 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448d3b4640 1 -- 172.21.15.23:0/1999160805 learned_addr learned my addr 172.21.15.23:0/1999160805 (peer_addr_for_me v2:172.21.15.23:0/0) 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448dbb5640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44881007f0 0x7f4488197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448e3b6640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 0x7f448819c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448d3b4640 1 -- 172.21.15.23:0/1999160805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 msgr2=0x7f448819c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.584+0000 7f448d3b4640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 0x7f448819c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448d3b4640 1 -- 172.21.15.23:0/1999160805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44881007f0 msgr2=0x7f4488197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448d3b4640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44881007f0 0x7f4488197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448d3b4640 1 -- 172.21.15.23:0/1999160805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448819ce80 con 0x7f4488101190 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448e3b6640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 0x7f448819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448d3b4640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c920 tx=0x7f447c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c007c70 con 0x7f4488101190 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f447c007e10 con 0x7f4488101190 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448819d170 con 0x7f4488101190 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c018890 con 0x7f4488101190 2024-03-28T16:03:53.588 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.585+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44881a4a00 con 0x7f4488101190 2024-03-28T16:03:53.589 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.586+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4488101cb0 con 0x7f4488101190 2024-03-28T16:03:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.587+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 4 ==== mgrmap(e 25) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f447c018a50 con 0x7f4488101190 2024-03-28T16:03:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.587+0000 7f447affd640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 0x7f44740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-28T16:03:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.588+0000 7f448dbb5640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 0x7f44740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-28T16:03:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.588+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8754+0+0 (secure 0 0 0) 0x7f447c0cf010 con 0x7f4488101190 2024-03-28T16:03:53.590 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.588+0000 7f448dbb5640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 0x7f44740a4eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4484000c00 tx=0x7f448402df90 comp rx=0 tx=0).ready entity=mgr.15546 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-28T16:03:53.592 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.592+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f447c014030 con 0x7f4488101190 2024-03-28T16:03:53.775 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:53.775+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 --> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f44881a4d90 con 0x7f4488101190 2024-03-28T16:03:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:53 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:53 smithi117 ceph-mon[36002]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T16:03:53.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:53 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:53 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:53 smithi120 ceph-mon[36826]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T16:03:53.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:53 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:53.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:53 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/2677195979' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:53.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:53 smithi023 ceph-mon[32448]: osdmap e105: 12 total, 12 up, 12 in 2024-03-28T16:03:53.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:53 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:54.456 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.456+0000 7f447affd640 1 -- 172.21.15.23:0/1999160805 <== mon.0 v2:172.21.15.23:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) v1 ==== 178+0+0 (secure 0 0 0) 0x7f447c09bc80 con 0x7f4488101190 2024-03-28T16:03:54.456 INFO:teuthology.orchestra.run.smithi023.stderr:pool 'cephfs_data_ec' removed 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 msgr2=0x7f44740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 0x7f44740a4eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4484000c00 tx=0x7f448402df90 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 msgr2=0x7f4488198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f447c00c920 tx=0x7f447c00cde0 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 shutdown_connections 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:6800/1692916537,v1:172.21.15.23:6801/1692916537] conn(0x7f44740a29f0 0x7f44740a4eb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.120:3300/0,v1:172.21.15.120:6789/0] conn(0x7f44881065d0 0x7f448819c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.23:3300/0,v1:172.21.15.23:6789/0] conn(0x7f4488101190 0x7f4488198420 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 --2- 172.21.15.23:0/1999160805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44881007f0 0x7f4488197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-28T16:03:54.460 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.459+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 >> 172.21.15.23:0/1999160805 conn(0x7f4488077340 msgr2=0x7f44880ffa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-28T16:03:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.460+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 shutdown_connections 2024-03-28T16:03:54.461 INFO:teuthology.orchestra.run.smithi023.stderr:2024-03-28T16:03:54.460+0000 7f448fe19640 1 -- 172.21.15.23:0/1999160805 wait complete. 2024-03-28T16:03:54.478 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:03:54.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:03:54.544 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:03:54.581 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-28T16:03:54.581 DEBUG:teuthology.orchestra.run.smithi023:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:03:54.598 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:03:54.619 DEBUG:teuthology.orchestra.run.smithi120:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-28T16:03:54.638 INFO:tasks.cephadm:Stopping all daemons... 2024-03-28T16:03:54.639 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-28T16:03:54.639 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a 2024-03-28T16:03:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:54 smithi117 ceph-mon[36002]: pgmap v337: 97 pgs: 97 active+clean; 577 KiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:03:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:54 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:54.821 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:54 smithi117 ceph-mon[36002]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T16:03:54.822 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:54 smithi117 ceph-mon[36002]: from='client.? 172.21.15.23:0/1999160805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:54 smithi120 ceph-mon[36826]: pgmap v337: 97 pgs: 97 active+clean; 577 KiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:03:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:54 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:54 smithi120 ceph-mon[36826]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T16:03:54.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:54 smithi120 ceph-mon[36826]: from='client.? 172.21.15.23:0/1999160805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-mon[32448]: pgmap v337: 97 pgs: 97 active+clean; 577 KiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-28T16:03:54.920 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/33758195' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-28T16:03:54.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-mon[32448]: osdmap e106: 12 total, 12 up, 12 in 2024-03-28T16:03:54.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-mon[32448]: from='client.? 172.21.15.23:0/1999160805' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-28T16:03:54.921 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 systemd[1]: Stopping Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:03:55.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a[32444]: 2024-03-28T16:03:54.953+0000 7f571eb6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:03:55.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a[32444]: 2024-03-28T16:03:54.953+0000 7f571eb6b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-28T16:03:55.280 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:54 smithi023 podman[92176]: 2024-03-28 16:03:54.974235179 +0000 UTC m=+0.147449830 container died 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T16:03:55.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 podman[92176]: 2024-03-28 16:03:55.40145549 +0000 UTC m=+0.574670199 container cleanup 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T16:03:55.639 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 bash[92176]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a 2024-03-28T16:03:55.879 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service' 2024-03-28T16:03:55.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 podman[92190]: 2024-03-28 16:03:55.639230755 +0000 UTC m=+0.660943087 container remove 92d871696cdcf8456e2f7d5e133659a156788f5e38b8e7cb42b9475c43cbb216 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-28T16:03:55.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-28T16:03:55.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 systemd[1]: Stopped Ceph mon.a for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T16:03:55.912 INFO:journalctl@ceph.mon.a.smithi023.stdout:Mar 28 16:03:55 smithi023 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.a.service: Consumed 18.451s CPU time. 2024-03-28T16:03:55.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:03:55.946 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-28T16:03:55.946 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-28T16:03:55.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.b 2024-03-28T16:03:56.262 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:55 smithi117 systemd[1]: Stopping Ceph mon.b for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:03:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b[35998]: 2024-03-28T16:03:56.262+0000 7f1ea0ddb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:03:56.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b[35998]: 2024-03-28T16:03:56.262+0000 7f1ea0ddb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-28T16:03:56.572 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:56 smithi117 podman[71685]: 2024-03-28 16:03:56.446284975 +0000 UTC m=+0.314524983 container died 7996064eeb221c6a6cb92c22548c3378a8f1e4a34a81954fc113d00dc9307965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T16:03:57.014 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:56 smithi117 podman[71685]: 2024-03-28 16:03:56.78752163 +0000 UTC m=+0.655761602 container cleanup 7996064eeb221c6a6cb92c22548c3378a8f1e4a34a81954fc113d00dc9307965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T16:03:57.014 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:56 smithi117 bash[71685]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b 2024-03-28T16:03:57.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.b.service' 2024-03-28T16:03:57.287 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:57 smithi117 podman[71699]: 2024-03-28 16:03:57.014060038 +0000 UTC m=+0.564187233 container remove 7996064eeb221c6a6cb92c22548c3378a8f1e4a34a81954fc113d00dc9307965 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T16:03:57.288 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:57 smithi117 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-28T16:03:57.288 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:57 smithi117 systemd[1]: Stopped Ceph mon.b for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T16:03:57.288 INFO:journalctl@ceph.mon.b.smithi117.stdout:Mar 28 16:03:57 smithi117 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.b.service: Consumed 10.389s CPU time. 2024-03-28T16:03:57.319 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:03:57.320 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-28T16:03:57.320 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-28T16:03:57.320 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.c 2024-03-28T16:03:57.656 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:57 smithi120 systemd[1]: Stopping Ceph mon.c for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:03:57.921 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c[36822]: 2024-03-28T16:03:57.655+0000 7f883e449700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:03:57.921 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c[36822]: 2024-03-28T16:03:57.655+0000 7f883e449700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-28T16:03:58.234 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:57 smithi120 podman[72660]: 2024-03-28 16:03:57.921372178 +0000 UTC m=+0.395826485 container died fe6917ceb952cae3c5c2e714e3b5bf0de41fd1da979824fd6dbb480571f9b44d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:03:58.585 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 podman[72660]: 2024-03-28 16:03:58.350491464 +0000 UTC m=+0.824945768 container cleanup fe6917ceb952cae3c5c2e714e3b5bf0de41fd1da979824fd6dbb480571f9b44d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-28T16:03:58.585 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 bash[72660]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c 2024-03-28T16:03:58.817 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.c.service' 2024-03-28T16:03:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 podman[72674]: 2024-03-28 16:03:58.585400597 +0000 UTC m=+0.659343378 container remove fe6917ceb952cae3c5c2e714e3b5bf0de41fd1da979824fd6dbb480571f9b44d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-28T16:03:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-28T16:03:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 systemd[1]: Stopped Ceph mon.c for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T16:03:58.839 INFO:journalctl@ceph.mon.c.smithi120.stdout:Mar 28 16:03:58 smithi120 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mon.c.service: Consumed 11.358s CPU time. 2024-03-28T16:03:58.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:03:58.874 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-28T16:03:58.874 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-28T16:03:58.874 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x 2024-03-28T16:03:59.170 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:58 smithi023 systemd[1]: Stopping Ceph mgr.x for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:03:59.519 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T16:03:59.191+0000 7ff174848700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:03:59.520 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x[32675]: 2024-03-28T16:03:59.191+0000 7ff174848700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-28T16:03:59.520 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 podman[92300]: 2024-03-28 16:03:59.223055139 +0000 UTC m=+0.154249854 container died ebed96533914fc7638821327fab3af2ccd05b009ac649a6aca7b23d3865a120d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-28T16:03:59.853 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 podman[92300]: 2024-03-28 16:03:59.626830633 +0000 UTC m=+0.558025348 container cleanup ebed96533914fc7638821327fab3af2ccd05b009ac649a6aca7b23d3865a120d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-28T16:03:59.854 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 bash[92300]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x 2024-03-28T16:04:00.104 INFO:journalctl@ceph.mgr.x.smithi023.stdout:Mar 28 16:03:59 smithi023 podman[92314]: 2024-03-28 16:03:59.853674902 +0000 UTC m=+0.627159328 container remove ebed96533914fc7638821327fab3af2ccd05b009ac649a6aca7b23d3865a120d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T16:04:00.107 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-28T16:04:00.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:00.145 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-28T16:04:00.145 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-28T16:04:00.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.y 2024-03-28T16:04:00.496 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 16:04:00 smithi117 systemd[1]: Stopping Ceph mgr.y for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:00.777 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 16:04:00 smithi117 podman[71805]: 2024-03-28 16:04:00.495985566 +0000 UTC m=+0.149479769 container died f9136aa59a52b65cb14ba8611967b4982786fc97b0b21ff0a7480c201709ddb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-28T16:04:01.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 16:04:00 smithi117 podman[71805]: 2024-03-28 16:04:00.875826205 +0000 UTC m=+0.529320401 container cleanup f9136aa59a52b65cb14ba8611967b4982786fc97b0b21ff0a7480c201709ddb4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-28T16:04:01.071 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Mar 28 16:04:00 smithi117 bash[71805]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-y 2024-03-28T16:04:01.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-28T16:04:01.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:01.326 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-28T16:04:01.327 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-28T16:04:01.327 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.z 2024-03-28T16:04:01.668 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 16:04:01 smithi120 systemd[1]: Stopping Ceph mgr.z for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:01.956 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 16:04:01 smithi120 podman[72780]: 2024-03-28 16:04:01.667866393 +0000 UTC m=+0.146929013 container died dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:02.290 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 16:04:02 smithi120 podman[72780]: 2024-03-28 16:04:02.030515103 +0000 UTC m=+0.509577730 container cleanup dd81e7516a46b70a225311645f5b2413f938c218a158dddd371520e09c67864f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:04:02.290 INFO:journalctl@ceph.mgr.z.smithi120.stdout:Mar 28 16:04:02 smithi120 bash[72780]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mgr-z 2024-03-28T16:04:02.506 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-28T16:04:02.544 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:02.545 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-28T16:04:02.545 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-28T16:04:02.545 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.0 2024-03-28T16:04:02.873 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:02 smithi023 systemd[1]: Stopping Ceph osd.0 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:03.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0[47228]: 2024-03-28T16:04:02.873+0000 7f215d205700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:03.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0[47228]: 2024-03-28T16:04:02.873+0000 7f215d205700 -1 osd.0 107 *** Got signal Terminated *** 2024-03-28T16:04:03.170 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0[47228]: 2024-03-28T16:04:02.873+0000 7f215d205700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:08.671 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:08 smithi023 podman[92424]: 2024-03-28 16:04:08.384896417 +0000 UTC m=+5.638327367 container died fa1e045cbcb2e8af3311a4ec38491a1296da47391b4784bc3f63730eca74872b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-03-28T16:04:09.150 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:08 smithi023 podman[92424]: 2024-03-28 16:04:08.881856099 +0000 UTC m=+6.135287048 container cleanup fa1e045cbcb2e8af3311a4ec38491a1296da47391b4784bc3f63730eca74872b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:09.150 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:08 smithi023 bash[92424]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0 2024-03-28T16:04:09.420 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92442]: 2024-03-28 16:04:09.149984805 +0000 UTC m=+0.756007446 container remove fa1e045cbcb2e8af3311a4ec38491a1296da47391b4784bc3f63730eca74872b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:09.681 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92530]: 2024-03-28 16:04:09.407271644 +0000 UTC m=+0.017645925 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:09.681 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92530]: 2024-03-28 16:04:09.542685566 +0000 UTC m=+0.153059833 container create e6a0c3668009afe53e50b6076ecd80eeb963c3d98f626f74c58edc407ef9907e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:04:10.097 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92530]: 2024-03-28 16:04:09.795220125 +0000 UTC m=+0.405594394 container init e6a0c3668009afe53e50b6076ecd80eeb963c3d98f626f74c58edc407ef9907e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:10.097 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92530]: 2024-03-28 16:04:09.7982985 +0000 UTC m=+0.408672772 container start e6a0c3668009afe53e50b6076ecd80eeb963c3d98f626f74c58edc407ef9907e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:10.097 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:09 smithi023 podman[92530]: 2024-03-28 16:04:09.861893209 +0000 UTC m=+0.472267476 container attach e6a0c3668009afe53e50b6076ecd80eeb963c3d98f626f74c58edc407ef9907e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T16:04:10.420 INFO:journalctl@ceph.osd.0.smithi023.stdout:Mar 28 16:04:10 smithi023 podman[92530]: 2024-03-28 16:04:10.097538527 +0000 UTC m=+0.707912794 container died e6a0c3668009afe53e50b6076ecd80eeb963c3d98f626f74c58edc407ef9907e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:04:10.724 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.0.service' 2024-03-28T16:04:10.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:10.768 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-28T16:04:10.769 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-28T16:04:10.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.1 2024-03-28T16:04:11.071 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:10 smithi117 systemd[1]: Stopping Ceph osd.1 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:11.571 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1[40568]: 2024-03-28T16:04:11.086+0000 7f1d53920700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:11.571 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1[40568]: 2024-03-28T16:04:11.086+0000 7f1d53920700 -1 osd.1 107 *** Got signal Terminated *** 2024-03-28T16:04:11.572 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1[40568]: 2024-03-28T16:04:11.086+0000 7f1d53920700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:16.780 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:16 smithi117 podman[71930]: 2024-03-28 16:04:16.478780886 +0000 UTC m=+5.514615875 container died 160386827f69e8365bcea1d59d876823bde7dc0be198e153734aa041f9c78892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-28T16:04:17.071 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:16 smithi117 podman[71930]: 2024-03-28 16:04:16.862696132 +0000 UTC m=+5.898531130 container cleanup 160386827f69e8365bcea1d59d876823bde7dc0be198e153734aa041f9c78892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-28T16:04:17.071 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:16 smithi117 bash[71930]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1 2024-03-28T16:04:17.438 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[71948]: 2024-03-28 16:04:17.131010133 +0000 UTC m=+0.648875930 container remove 160386827f69e8365bcea1d59d876823bde7dc0be198e153734aa041f9c78892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:04:17.701 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.341982529 +0000 UTC m=+0.018065516 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:17.702 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.44866467 +0000 UTC m=+0.124747645 container create 3284ceb93999327b6a089449506d7d228a5bcfe5303be4801a41cf7b5a59f21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:17.702 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.701031029 +0000 UTC m=+0.377114007 container init 3284ceb93999327b6a089449506d7d228a5bcfe5303be4801a41cf7b5a59f21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:17.997 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.704383724 +0000 UTC m=+0.380466708 container start 3284ceb93999327b6a089449506d7d228a5bcfe5303be4801a41cf7b5a59f21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-28T16:04:17.998 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.801255078 +0000 UTC m=+0.477338073 container attach 3284ceb93999327b6a089449506d7d228a5bcfe5303be4801a41cf7b5a59f21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:04:18.262 INFO:journalctl@ceph.osd.1.smithi117.stdout:Mar 28 16:04:17 smithi117 podman[72041]: 2024-03-28 16:04:17.997904868 +0000 UTC m=+0.673987851 container died 3284ceb93999327b6a089449506d7d228a5bcfe5303be4801a41cf7b5a59f21d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:18.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.1.service' 2024-03-28T16:04:18.728 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:18.729 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-28T16:04:18.729 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-28T16:04:18.729 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.2 2024-03-28T16:04:19.043 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:18 smithi120 systemd[1]: Stopping Ceph osd.2 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:19.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2[41989]: 2024-03-28T16:04:19.042+0000 7f39002df700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:19.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2[41989]: 2024-03-28T16:04:19.042+0000 7f39002df700 -1 osd.2 107 *** Got signal Terminated *** 2024-03-28T16:04:19.339 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2[41989]: 2024-03-28T16:04:19.042+0000 7f39002df700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:24.839 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:24 smithi120 podman[72905]: 2024-03-28 16:04:24.580869984 +0000 UTC m=+5.659097523 container died 5650db16f7c6d1c6e2b81da4f7401b744080f11f38f676b6f10d1bb73310a0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:25.193 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[72905]: 2024-03-28 16:04:25.002759086 +0000 UTC m=+6.080986628 container cleanup 5650db16f7c6d1c6e2b81da4f7401b744080f11f38f676b6f10d1bb73310a0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:04:25.193 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 bash[72905]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2 2024-03-28T16:04:25.556 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[72922]: 2024-03-28 16:04:25.254619087 +0000 UTC m=+0.665163036 container remove 5650db16f7c6d1c6e2b81da4f7401b744080f11f38f676b6f10d1bb73310a0ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-28T16:04:25.840 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[73021]: 2024-03-28 16:04:25.459188613 +0000 UTC m=+0.017787873 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:25.840 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[73021]: 2024-03-28 16:04:25.597114978 +0000 UTC m=+0.155714215 container create c6e994dd6c5b2a6bcd3322cb2815a96695a7b442cb0d2cd8f42e2224895ed73b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T16:04:26.136 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[73021]: 2024-03-28 16:04:25.841824559 +0000 UTC m=+0.400423822 container init c6e994dd6c5b2a6bcd3322cb2815a96695a7b442cb0d2cd8f42e2224895ed73b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T16:04:26.136 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[73021]: 2024-03-28 16:04:25.844964172 +0000 UTC m=+0.403563417 container start c6e994dd6c5b2a6bcd3322cb2815a96695a7b442cb0d2cd8f42e2224895ed73b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-28T16:04:26.136 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:25 smithi120 podman[73021]: 2024-03-28 16:04:25.933616426 +0000 UTC m=+0.492215672 container attach c6e994dd6c5b2a6bcd3322cb2815a96695a7b442cb0d2cd8f42e2224895ed73b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:26.438 INFO:journalctl@ceph.osd.2.smithi120.stdout:Mar 28 16:04:26 smithi120 podman[73021]: 2024-03-28 16:04:26.136694769 +0000 UTC m=+0.695294013 container died c6e994dd6c5b2a6bcd3322cb2815a96695a7b442cb0d2cd8f42e2224895ed73b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T16:04:26.752 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.2.service' 2024-03-28T16:04:26.794 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:26.797 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-28T16:04:26.798 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-28T16:04:26.798 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.3 2024-03-28T16:04:27.118 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:26 smithi023 systemd[1]: Stopping Ceph osd.3 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:27.420 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T16:04:27.117+0000 7fc63d39b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:27.420 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T16:04:27.117+0000 7fc63d39b700 -1 osd.3 107 *** Got signal Terminated *** 2024-03-28T16:04:27.420 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:27 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T16:04:27.117+0000 7fc63d39b700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:30.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:29 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:29.889+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:30.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:30.082+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:30.920 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:30 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T16:04:30.478+0000 7fc635987700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:04.655913+0000 front 2024-03-28T16:04:04.655946+0000 (oldest deadline 2024-03-28T16:04:29.956027+0000) 2024-03-28T16:04:31.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:30 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:30.906+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:31.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:31.059+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:31.920 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:31 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3[53924]: 2024-03-28T16:04:31.463+0000 7fc635987700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:04.655913+0000 front 2024-03-28T16:04:04.655946+0000 (oldest deadline 2024-03-28T16:04:29.956027+0000) 2024-03-28T16:04:32.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:31 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:31.894+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:32.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:32.080+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:32.944 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:32 smithi023 podman[92637]: 2024-03-28 16:04:32.671783818 +0000 UTC m=+5.677811413 container died 6d4e984c08704719b348e6940649cd3902e2d7f5fc04fc977895792f54f915ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T16:04:33.202 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92637]: 2024-03-28 16:04:33.034790932 +0000 UTC m=+6.040818516 container cleanup 6d4e984c08704719b348e6940649cd3902e2d7f5fc04fc977895792f54f915ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:33.203 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 bash[92637]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3 2024-03-28T16:04:33.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:32 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:32.875+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:33.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:33.038+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:33.663 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92665]: 2024-03-28 16:04:33.311391827 +0000 UTC m=+0.643243247 container remove 6d4e984c08704719b348e6940649cd3902e2d7f5fc04fc977895792f54f915ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-28T16:04:33.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:33.457+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:33.920 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92753]: 2024-03-28 16:04:33.56670417 +0000 UTC m=+0.017830374 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:33.920 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92753]: 2024-03-28 16:04:33.687636067 +0000 UTC m=+0.138762310 container create 6f5537dc5e83a45b4df1ecec782892e2b10ce19688cbe8336f6d1b869aab9137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T16:04:34.243 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92753]: 2024-03-28 16:04:33.940195085 +0000 UTC m=+0.391321290 container init 6f5537dc5e83a45b4df1ecec782892e2b10ce19688cbe8336f6d1b869aab9137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:34.243 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:33 smithi023 podman[92753]: 2024-03-28 16:04:33.943326177 +0000 UTC m=+0.394452386 container start 6f5537dc5e83a45b4df1ecec782892e2b10ce19688cbe8336f6d1b869aab9137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:34.243 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:34 smithi023 podman[92753]: 2024-03-28 16:04:34.040143352 +0000 UTC m=+0.491269576 container attach 6f5537dc5e83a45b4df1ecec782892e2b10ce19688cbe8336f6d1b869aab9137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-28T16:04:34.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:33 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:33.865+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:34.340 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:34.088+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:34.554 INFO:journalctl@ceph.osd.3.smithi023.stdout:Mar 28 16:04:34 smithi023 podman[92753]: 2024-03-28 16:04:34.242850894 +0000 UTC m=+0.693977100 container died 6f5537dc5e83a45b4df1ecec782892e2b10ce19688cbe8336f6d1b869aab9137 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T16:04:34.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:34.412+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:34.860 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.3.service' 2024-03-28T16:04:34.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:34.910 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-28T16:04:34.910 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-28T16:04:34.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.4 2024-03-28T16:04:35.224 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:34 smithi117 systemd[1]: Stopping Ceph osd.4 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:35.225 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:34 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:34.839+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:35.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:35.057+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:35.571 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:35 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:35.224+0000 7fb653654700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:35.571 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:35 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:35.224+0000 7fb653654700 -1 osd.4 107 *** Got signal Terminated *** 2024-03-28T16:04:35.571 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:35 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:35.224+0000 7fb653654700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:35.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:35.383+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:36.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:35 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:35.874+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:36.338 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:36 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:36.056+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:36.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:36 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:36.337+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:37.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:36 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:36.884+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:37.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:36 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:36.862+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:37.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:37 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:37.038+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:37.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:37 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:37.350+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:38.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:37 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:37.922+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:38.322 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:38 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:38.040+0000 7fb64fc48700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:16.037307+0000 front 2024-03-28T16:04:16.037334+0000 (oldest deadline 2024-03-28T16:04:37.737444+0000) 2024-03-28T16:04:38.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:37 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:37.847+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:38.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:38 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:38.042+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:38.832 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:38 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:38.340+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:38.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:38 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:38.499+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:39.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:38 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:38.831+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:39.321 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:39 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:39.019+0000 7fb64fc48700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:16.037307+0000 front 2024-03-28T16:04:16.037334+0000 (oldest deadline 2024-03-28T16:04:37.737444+0000) 2024-03-28T16:04:39.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:38 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:38.947+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:39.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:38 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:38.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:39.390 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:39 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:39.089+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:39.735 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:39 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:39.089+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:39.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:39 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:39.389+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:39.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:39 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:39.511+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:39.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:39 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:39.452+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:40.321 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:39 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4[47111]: 2024-03-28T16:04:39.996+0000 7fb64fc48700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:16.037307+0000 front 2024-03-28T16:04:16.037334+0000 (oldest deadline 2024-03-28T16:04:37.737444+0000) 2024-03-28T16:04:40.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:39 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:39.900+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:40.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:39 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:39.922+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:40.339 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:39 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:39.872+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:40.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:40 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:40.090+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:40.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:40 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:40.090+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:40.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:40 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:40.405+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:40.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:40 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:40.405+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:40.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:40 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:40.436+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:40.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:40 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:40.542+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:41.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:40 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:40.900+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:41.071 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:40 smithi117 podman[72164]: 2024-03-28 16:04:40.780909081 +0000 UTC m=+5.680045959 container died 9db1668ece353f786a69505467f3de9e07ee8725746c28bb4753b1401dfe054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:04:41.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:40 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:40.948+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:41.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:40 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:40.948+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:41.136 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:40 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:40.879+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:41.433 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:41 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:41.135+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:41.433 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:41 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:41.135+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:41.441 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:41 smithi117 podman[72164]: 2024-03-28 16:04:41.16492392 +0000 UTC m=+6.064060799 container cleanup 9db1668ece353f786a69505467f3de9e07ee8725746c28bb4753b1401dfe054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-28T16:04:41.441 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:41 smithi117 bash[72164]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4 2024-03-28T16:04:41.670 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:41 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:41.398+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:41.670 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:41 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:41.523+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:41.775 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:41 smithi117 podman[72183]: 2024-03-28 16:04:41.441443626 +0000 UTC m=+0.659607576 container remove 9db1668ece353f786a69505467f3de9e07ee8725746c28bb4753b1401dfe054c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-28T16:04:41.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:41 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:41.432+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:41.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:41 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:41.432+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:42.069 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:41 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:41.867+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:42.070 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:41 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:41.942+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:42.070 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:41 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:41.942+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:42.070 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:41 smithi117 podman[72273]: 2024-03-28 16:04:41.775484278 +0000 UTC m=+0.111334306 container create adf76397832dd37e7ca72d324106784f60283fb0e3e60c776c00f28b9b1f1575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:04:42.070 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:41 smithi117 podman[72273]: 2024-03-28 16:04:41.682385644 +0000 UTC m=+0.018235682 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:42.070 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:42 smithi117 podman[72273]: 2024-03-28 16:04:42.06954819 +0000 UTC m=+0.405398219 container init adf76397832dd37e7ca72d324106784f60283fb0e3e60c776c00f28b9b1f1575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-28T16:04:42.184 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:41 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:41.883+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:42.321 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:42 smithi117 podman[72273]: 2024-03-28 16:04:42.073748018 +0000 UTC m=+0.409598046 container start adf76397832dd37e7ca72d324106784f60283fb0e3e60c776c00f28b9b1f1575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T16:04:42.321 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:42 smithi117 podman[72273]: 2024-03-28 16:04:42.169715668 +0000 UTC m=+0.505565698 container attach adf76397832dd37e7ca72d324106784f60283fb0e3e60c776c00f28b9b1f1575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True) 2024-03-28T16:04:42.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:42 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:42.397+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:42.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:42 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:42.397+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:42.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:42 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:42.183+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:42.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:42 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:42.183+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:42.739 INFO:journalctl@ceph.osd.4.smithi117.stdout:Mar 28 16:04:42 smithi117 podman[72273]: 2024-03-28 16:04:42.368853894 +0000 UTC m=+0.704703923 container died adf76397832dd37e7ca72d324106784f60283fb0e3e60c776c00f28b9b1f1575 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T16:04:42.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:42 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:42.437+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:42.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:42 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:42.531+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:43.032 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:42 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:42.844+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:43.032 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:42 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:42.938+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:43.032 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:42 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:42.938+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:43.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.4.service' 2024-03-28T16:04:43.108 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:43.108 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-28T16:04:43.108 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-28T16:04:43.109 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.5 2024-03-28T16:04:43.138 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:42 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:42.875+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:43.424 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:43.366+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:43.366+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 systemd[1]: Stopping Ceph osd.5 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:43.217+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:43.217+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:43.423+0000 7fdb90fe6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:43.423+0000 7fdb90fe6700 -1 osd.5 107 *** Got signal Terminated *** 2024-03-28T16:04:43.425 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:43.423+0000 7fdb90fe6700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:43.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:43 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:43.446+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:43.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:43 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:43.560+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:44.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:43 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:43.799+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:44.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:43 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:43.941+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:44.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:43 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:43.941+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:44.184 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:43 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:43.873+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:44.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:44.184+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:44.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:44.184+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:44.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:44.364+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:44.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:44.364+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:44.610 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:44 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:44.474+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:44.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:44 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:44.610+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:45.072 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:44 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:44.768+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:45.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:44 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:44.978+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:45.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:44 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:44.978+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:45.339 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:44.919+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:45.339 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:44 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:44.919+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:45.339 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:45.166+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:45.340 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:45.166+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:45.839 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:45.342+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:45.840 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:45.342+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:45.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:45 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:45.515+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:45.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:45 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:45.607+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:46.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:45 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:45.952+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:46.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:45 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:45.952+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:46.072 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:45 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:45.814+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:46.072 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:45 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:45.814+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:46.208 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:45.944+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:46.209 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:45 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:45.944+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:46.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:46.208+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:46.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:46.208+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:46.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:46.301+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:46.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:46.301+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:46.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:46.301+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:46.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:46.558+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:46.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:46.638+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:46.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:46.638+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:47.197 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:46.907+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:47.197 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:46 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:46.907+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:47.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:46.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:47.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:46.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:47.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:46.846+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:47.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:46 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:46.846+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:47.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:47.197+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:47.589 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:47.197+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:47.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:47.197+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.644217+0000 front 2024-03-28T16:04:18.644161+0000 (oldest deadline 2024-03-28T16:04:46.244804+0000) 2024-03-28T16:04:47.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:47.253+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:47.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:47.253+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:47.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:47.253+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:47.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:47.573+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:47.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:47.616+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:47.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:47 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:47.616+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:48.211 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:47.878+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:48.212 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:47 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:47.878+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:48.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:47 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:47.915+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:48.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:47 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:47.915+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:48.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:47 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:47.855+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:48.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:47 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:47.855+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:48.211+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:07.642425+0000 front 2024-03-28T16:04:07.642285+0000 (oldest deadline 2024-03-28T16:04:29.342178+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:48.211+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:12.843315+0000 front 2024-03-28T16:04:12.843232+0000 (oldest deadline 2024-03-28T16:04:38.643783+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5[48508]: 2024-03-28T16:04:48.211+0000 7fdb895d2700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.644217+0000 front 2024-03-28T16:04:18.644161+0000 (oldest deadline 2024-03-28T16:04:46.244804+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:48.234+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:48.234+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:48.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:48.234+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:48.922 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:48.565+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:48.923 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:48.597+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:48.923 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:48 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:48.597+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:49.133 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:48.884+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:49.133 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:48 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:48.884+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:49.134 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:48 smithi120 podman[73129]: 2024-03-28 16:04:48.841891429 +0000 UTC m=+5.543112793 container died 126a19d823fbf6873ae87707fd560a1a75185c501b6c1e8f24de1f596823bcb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-28T16:04:49.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:48.888+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:49.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:48.888+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:49.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:48.847+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:49.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:48 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:48.847+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:49.500 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:49 smithi120 podman[73129]: 2024-03-28 16:04:49.23279685 +0000 UTC m=+5.934018214 container cleanup 126a19d823fbf6873ae87707fd560a1a75185c501b6c1e8f24de1f596823bcb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:04:49.501 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:49 smithi120 bash[73129]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5 2024-03-28T16:04:49.501 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:49.252+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:49.501 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:49.253+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:49.501 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:49.253+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:49.840 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:49 smithi120 podman[73147]: 2024-03-28 16:04:49.500945401 +0000 UTC m=+0.652453486 container remove 126a19d823fbf6873ae87707fd560a1a75185c501b6c1e8f24de1f596823bcb0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T16:04:49.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:49.606+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:49.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:49.606+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:49.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:49.576+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:49.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:49 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:49.576+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:50.138 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:49 smithi120 podman[73234]: 2024-03-28 16:04:49.743403032 +0000 UTC m=+0.017793011 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:50.138 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:49 smithi120 podman[73234]: 2024-03-28 16:04:49.868387305 +0000 UTC m=+0.142777269 container create e484bf560c6cc3d53952eb91a1a0847562173e75e56459655c807b809b3f29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:50.138 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:50 smithi120 podman[73234]: 2024-03-28 16:04:50.137707813 +0000 UTC m=+0.412097781 container init e484bf560c6cc3d53952eb91a1a0847562173e75e56459655c807b809b3f29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-28T16:04:50.138 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:49.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:50.138 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:49 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:49.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:50.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:49.884+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:50.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:49.884+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:50.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:49.889+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:50.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:49.889+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:50.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:49 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:49.889+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:50.428 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:50.263+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:50.429 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:50.263+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:50.429 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:50.263+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:50.429 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:50 smithi120 podman[73234]: 2024-03-28 16:04:50.141113747 +0000 UTC m=+0.415503713 container start e484bf560c6cc3d53952eb91a1a0847562173e75e56459655c807b809b3f29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:04:50.429 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:50 smithi120 podman[73234]: 2024-03-28 16:04:50.237633129 +0000 UTC m=+0.512023107 container attach e484bf560c6cc3d53952eb91a1a0847562173e75e56459655c807b809b3f29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-28T16:04:50.732 INFO:journalctl@ceph.osd.5.smithi120.stdout:Mar 28 16:04:50 smithi120 podman[73234]: 2024-03-28 16:04:50.429099106 +0000 UTC m=+0.703489087 container died e484bf560c6cc3d53952eb91a1a0847562173e75e56459655c807b809b3f29eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True) 2024-03-28T16:04:50.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:50.616+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:50.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:50.616+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:50.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:50.543+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:50.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:50 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:50.543+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:51.041 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:50.814+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:51.041 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:50 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:50.814+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:51.064 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.5.service' 2024-03-28T16:04:51.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:51.150 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-28T16:04:51.150 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-28T16:04:51.150 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.6 2024-03-28T16:04:51.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:50.920+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:51.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:50.920+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:51.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:50.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:51.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:50.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:51.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:50 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:50.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:51.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:51.220+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:51.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:51.220+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:51.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:51.220+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:51.472 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 systemd[1]: Stopping Ceph osd.6 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:51.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:51.538+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:51.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:51.538+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:51.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:51.470+0000 7effc897a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:51.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:51.470+0000 7effc897a700 -1 osd.6 107 *** Got signal Terminated *** 2024-03-28T16:04:51.921 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:51.470+0000 7effc897a700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:51.921 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:51.623+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:51.921 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:51.623+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:52.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:51.765+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:52.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:51 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:51.765+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:52.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:51.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:52.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:51.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:52.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:51.858+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:52.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:51.858+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:52.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:51 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:51.858+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:52.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:52 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:52.206+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:52.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:52 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:52.206+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:52.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:52 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:52.206+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:52.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:52.628+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:52.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:52.628+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:52.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:52.533+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:52.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:52 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:52.533+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:53.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:52 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:52.789+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:53.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:52 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:52.789+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:53.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:52.883+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:53.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:52.883+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:53.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:52.880+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:53.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:52.880+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:53.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:52.880+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:53.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:52 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:52.880+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:53.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:53 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:53.256+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:53.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:53 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:53.256+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:53.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:53 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:53.256+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:53.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:53 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:53.672+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:53.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:53 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:53.672+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:53.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:53 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:53.570+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:53.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:53 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:53.570+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:54.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:53 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:53.795+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:54.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:53 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:53.795+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:54.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:53.930+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:54.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:53.930+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:54.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:53.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:54.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:53.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:54.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:53.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:54.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:53 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:53.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:54.368 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:54.228+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:54.368 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:54.228+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:54.368 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:54.228+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:54.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:54.561+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:54.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:54.561+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:54.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:54.706+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:54.921 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:54 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:54.706+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:55.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:54.822+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:55.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:54.822+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:55.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:54 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:54.822+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:04:55.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:54.956+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:55.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:54.956+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:55.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:54.935+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:55.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:54.935+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:55.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:54.935+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:55.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:54 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:54.935+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:55.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:55.202+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:55.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:55.202+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:55.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:55.202+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:55.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:55.706+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.691303+0000 front 2024-03-28T16:04:14.691248+0000 (oldest deadline 2024-03-28T16:04:39.391344+0000) 2024-03-28T16:04:55.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:55.706+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:23.492561+0000 front 2024-03-28T16:04:23.492604+0000 (oldest deadline 2024-03-28T16:04:49.392713+0000) 2024-03-28T16:04:55.920 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6[60448]: 2024-03-28T16:04:55.706+0000 7effc4f6e700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.893572+0000 front 2024-03-28T16:04:29.893750+0000 (oldest deadline 2024-03-28T16:04:55.193789+0000) 2024-03-28T16:04:55.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:55.593+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:55.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:55.593+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:55.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:55 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:55.593+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:04:56.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:55.826+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:56.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:55.826+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:56.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:55 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:55.826+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:55.910+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:55.910+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:55.927+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:55.927+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:55.927+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:56.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:55.927+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:56.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:56.165+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:56.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:56.165+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:56.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:56.165+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:56.858 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:56 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:56.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:56.858 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:56 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:56.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:56.858 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:56 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:56.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:04:57.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:56.777+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:57.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:56.777+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:57.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:56 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:56.777+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:04:57.130 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:56 smithi023 podman[92859]: 2024-03-28 16:04:56.858085475 +0000 UTC m=+5.513596889 container died 42236009995be1256ded6778f58f18cb8fb94dd5b1cdaf5bac2e5bc3150da6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T16:04:57.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:56.871+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:57.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:56.871+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:57.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:56.871+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:04:57.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:56.933+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:57.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:56.933+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:57.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:56.933+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:57.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:56 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:56.933+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:57.420 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:57 smithi023 podman[92859]: 2024-03-28 16:04:57.229942766 +0000 UTC m=+5.885454179 container cleanup 42236009995be1256ded6778f58f18cb8fb94dd5b1cdaf5bac2e5bc3150da6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-28T16:04:57.420 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:57 smithi023 bash[92859]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6 2024-03-28T16:04:57.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:57.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:57.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:57.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:57.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:57.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:57.890 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:57 smithi023 podman[92876]: 2024-03-28 16:04:57.581026381 +0000 UTC m=+0.717280445 container remove 42236009995be1256ded6778f58f18cb8fb94dd5b1cdaf5bac2e5bc3150da6ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T16:04:57.890 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:57 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:57.525+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:57.890 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:57 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:57.525+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:57.890 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:57 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:57.525+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:04:58.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:57.752+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:58.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:57.752+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:58.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:57 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:57.752+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:04:58.142 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:57 smithi023 podman[92963]: 2024-03-28 16:04:57.890210152 +0000 UTC m=+0.109818493 container create f279975303f8bb8ed73fafaf8305a32b116dbc7ed7a6eb76c05b3c94729a4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-28T16:04:58.142 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:57 smithi023 podman[92963]: 2024-03-28 16:04:57.798233998 +0000 UTC m=+0.017842333 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:04:58.142 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:58 smithi023 podman[92963]: 2024-03-28 16:04:58.142836343 +0000 UTC m=+0.362444677 container init f279975303f8bb8ed73fafaf8305a32b116dbc7ed7a6eb76c05b3c94729a4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-28T16:04:58.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:57.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:58.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:57.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:58.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:57.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:04:58.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:57.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:58.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:57.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:58.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:57.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:58.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:57 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:57.926+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:58.420 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:58 smithi023 podman[92963]: 2024-03-28 16:04:58.145941511 +0000 UTC m=+0.365549846 container start f279975303f8bb8ed73fafaf8305a32b116dbc7ed7a6eb76c05b3c94729a4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:04:58.421 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:58 smithi023 podman[92963]: 2024-03-28 16:04:58.242873107 +0000 UTC m=+0.462481443 container attach f279975303f8bb8ed73fafaf8305a32b116dbc7ed7a6eb76c05b3c94729a4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T16:04:58.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:58.178+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:58.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:58.178+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:58.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:58.178+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:58.904 INFO:journalctl@ceph.osd.6.smithi023.stdout:Mar 28 16:04:58 smithi023 podman[92963]: 2024-03-28 16:04:58.441414862 +0000 UTC m=+0.661023210 container died f279975303f8bb8ed73fafaf8305a32b116dbc7ed7a6eb76c05b3c94729a4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:04:58.904 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:58.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:58.904 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:58.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:58.904 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:58 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:58.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:04:59.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:58.711+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:04:59.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:58.711+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:04:59.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:58 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:58.711+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:04:59.195 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.6.service' 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:58.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:58.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:58.888+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:58.878+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:58.878+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:58.878+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:04:59.222 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:58 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:58.878+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:04:59.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:04:59.237 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-28T16:04:59.237 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-28T16:04:59.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.7 2024-03-28T16:04:59.553 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 systemd[1]: Stopping Ceph osd.7 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:04:59.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:59.172+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:04:59.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:59.172+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:04:59.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:04:59.172+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:04:59.821 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.566+0000 7f736066d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:04:59.821 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.566+0000 7f736066d700 -1 osd.7 107 *** Got signal Terminated *** 2024-03-28T16:04:59.821 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.566+0000 7f736066d700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:04:59.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:59.499+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:04:59.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:59.499+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:04:59.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:04:59 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:04:59.499+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:00.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:59.732+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:00.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:59.732+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:00.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:04:59 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:04:59.732+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:00.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.895+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:05:00.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.895+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:05:00.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:04:59.895+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:05:00.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:59.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:00.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:59.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:00.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:59.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:00.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:04:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:04:59.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:00.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:00.194+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:00.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:00.194+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:00.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:00.194+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:00.821 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:00.506+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:00.822 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:00.506+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:00.822 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:00 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:00.506+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:01.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:00.742+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:01.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:00.742+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:01.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:00 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:00.742+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:01.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:00.903+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:00.903+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:00.903+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:00.837+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:00.837+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:00.837+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:01.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:00 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:00.837+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:01.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:01.232+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:01.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:01.232+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:01.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:01.232+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:01.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:01.535+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:01.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:01.535+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:01.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:01 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:01.535+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:02.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:01.756+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:02.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:01.756+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:02.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:01 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:01.756+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:02.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:01.925+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:01.925+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:01.925+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:01.871+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:01.871+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:01.871+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:01.871+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:02.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:01 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:01.871+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:02.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:02.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:02.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:02.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:02.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:02.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:02.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:02.526+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:02.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:02.526+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:02.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:02 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:02.527+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:03.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:02.754+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:03.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:02.754+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:03.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:02.754+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:03.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:02 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:02.754+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:03.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:02.876+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:05:03.321 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:02.876+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:02.876+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:02.877+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:37.442633+0000 front 2024-03-28T16:04:37.442674+0000 (oldest deadline 2024-03-28T16:05:02.742806+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:02.835+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:02.835+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:02.835+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:02.835+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:03.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:02 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:02.835+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:03.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:03.196+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:03.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:03.196+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:03.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:03.196+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:03.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:03.196+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:03.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:03.542+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:03.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:03.542+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:03.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:03 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:03.542+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:04.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:03.867+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.238040+0000 front 2024-03-28T16:04:14.238006+0000 (oldest deadline 2024-03-28T16:04:36.538232+0000) 2024-03-28T16:05:04.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:03.867+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.039430+0000 front 2024-03-28T16:04:20.039584+0000 (oldest deadline 2024-03-28T16:04:45.339582+0000) 2024-03-28T16:05:04.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:03.867+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:31.041888+0000 front 2024-03-28T16:04:31.041809+0000 (oldest deadline 2024-03-28T16:04:56.341644+0000) 2024-03-28T16:05:04.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7[53695]: 2024-03-28T16:05:03.867+0000 7f7358c59700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:37.442633+0000 front 2024-03-28T16:04:37.442674+0000 (oldest deadline 2024-03-28T16:05:02.742806+0000) 2024-03-28T16:05:04.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:03.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:04.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:03.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:04.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:03.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:04.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:03.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:04.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:03.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:04.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:03.801+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:04.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:03.801+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:04.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:03.801+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:04.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:03 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:03.801+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:04.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:04.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:04.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:04.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:04.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:04.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:04.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:04.204+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:04.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:04.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:04.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:04.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:04.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:04.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:04.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:04 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:04.552+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:05.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:04 smithi117 podman[72379]: 2024-03-28 16:05:04.936438337 +0000 UTC m=+5.484866147 container died 960e9b90c15d9aa17f336b4f6e14b4a3c047e842e5619485b761ea7b9db22b43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:05.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:04.765+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:05.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:04.765+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:05.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:04.765+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:05.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:04.765+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:05.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:04 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:04.765+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:05.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:04.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:05.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:04.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:05.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:04.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:05.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:04 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:04.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:05.483 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:05 smithi117 podman[72379]: 2024-03-28 16:05:05.234380524 +0000 UTC m=+5.782808297 container cleanup 960e9b90c15d9aa17f336b4f6e14b4a3c047e842e5619485b761ea7b9db22b43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-28T16:05:05.484 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:05 smithi117 bash[72379]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7 2024-03-28T16:05:05.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:05.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:05.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:05.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:05.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:05.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:05.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:05.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:05.803 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:05 smithi117 podman[72406]: 2024-03-28 16:05:05.483692419 +0000 UTC m=+0.548846709 container remove 960e9b90c15d9aa17f336b4f6e14b4a3c047e842e5619485b761ea7b9db22b43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T16:05:05.804 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:05.727+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:05.804 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:05.727+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:05.804 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:05.727+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:05.804 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:05.727+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:05.804 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:05 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:05.727+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:05.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:05.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:05.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:05.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:05.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:05.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:05.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:05 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:05.502+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:06.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:05 smithi117 podman[72503]: 2024-03-28 16:05:05.70644267 +0000 UTC m=+0.017480032 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:05:06.071 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:05 smithi117 podman[72503]: 2024-03-28 16:05:05.834474358 +0000 UTC m=+0.145511724 container create 6df659536d9a0014ab20323d67f7258325b93280d17863a6da44706502b95d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319) 2024-03-28T16:05:06.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:05.818+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:06.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:05.818+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:06.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:05.818+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:06.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:05 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:05.818+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:06.381 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:06 smithi117 podman[72503]: 2024-03-28 16:05:06.090841296 +0000 UTC m=+0.401878645 container init 6df659536d9a0014ab20323d67f7258325b93280d17863a6da44706502b95d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T16:05:06.381 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:06 smithi117 podman[72503]: 2024-03-28 16:05:06.094378096 +0000 UTC m=+0.405415444 container start 6df659536d9a0014ab20323d67f7258325b93280d17863a6da44706502b95d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-28T16:05:06.381 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:06 smithi117 podman[72503]: 2024-03-28 16:05:06.17056835 +0000 UTC m=+0.481605713 container attach 6df659536d9a0014ab20323d67f7258325b93280d17863a6da44706502b95d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T16:05:06.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:06.142+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:06.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:06.142+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:06.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:06.142+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:06.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:06.142+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:06.689 INFO:journalctl@ceph.osd.7.smithi117.stdout:Mar 28 16:05:06 smithi117 podman[72503]: 2024-03-28 16:05:06.382023076 +0000 UTC m=+0.693060443 container died 6df659536d9a0014ab20323d67f7258325b93280d17863a6da44706502b95d4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:05:06.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:06 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:06.549+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:06.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:06 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:06.549+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:06.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:06 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:06.549+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:06.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:06 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:06.549+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:06.957 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:06 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:06.688+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:06.957 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:06 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:06.688+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:06.957 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:06 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:06.688+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:06.957 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:06 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:06.688+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:06.957 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:06 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:06.688+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:06.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.7.service' 2024-03-28T16:05:07.020 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:07.021 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-28T16:05:07.021 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-28T16:05:07.021 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.8 2024-03-28T16:05:07.052 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:06.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:07.052 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:06.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:07.052 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:06.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:07.052 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:06 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:06.790+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:07.339 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 systemd[1]: Stopping Ceph osd.8 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:07.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:07.157+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:07.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:07.157+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:07.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:07.157+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:07.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:07.157+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:07.816 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.343+0000 7fb6727de700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.343+0000 7fb6727de700 -1 osd.8 107 *** Got signal Terminated *** 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.343+0000 7fb6727de700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.815+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.815+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.815+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:07.817 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:07 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:07.815+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:07.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:07 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:07.527+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:07.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:07 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:07.527+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:07.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:07 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:07.527+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:07.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:07 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:07.527+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:08.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:07.723+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:08.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:07.723+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:08.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:07.723+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:08.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:07.723+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:08.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:07.723+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:08.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:08.197+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:08.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:08.197+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:08.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:08.197+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:08.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:08.197+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:08.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:08 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:08.486+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:08.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:08 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:08.486+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:08.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:08 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:08.486+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:08.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:08 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:08.486+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:09.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:08.747+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:09.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:08.747+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:09.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:08.747+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:09.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:08.747+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:09.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:08 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:08.747+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:09.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:08.771+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:09.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:08.771+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:09.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:08.771+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:09.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:08 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:08.771+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:09.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:09.158+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:09.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:09.158+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:09.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:09.158+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:09.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:09.158+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:09.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:09 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:09.510+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:09.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:09 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:09.510+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:09.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:09 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:09.510+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:09.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:09 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:09.510+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:09.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:09 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:09.510+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:10.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:10.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:10.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:10.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:10.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:10.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:09 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:09.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:10.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:09.812+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:10.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:09.812+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:10.089 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:09.812+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:10.090 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:09 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:09.812+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:10.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:10.137+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:10.633 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:10.137+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:10.633 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:10.137+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:10.633 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:10.137+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:10.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:10.467+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:10.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:10.467+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:10.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:10.467+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:10.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:10.467+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:10.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:10 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:10.467+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:11.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:10 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:10.758+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:11.120 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:10.849+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:11.120 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:10.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:11.120 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:10.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:11.120 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:10.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:11.121 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:10 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:10.850+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.146748+0000 front 2024-03-28T16:04:46.146696+0000 (oldest deadline 2024-03-28T16:05:10.246767+0000) 2024-03-28T16:05:11.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:11.119+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:11.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:11.119+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:11.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:11.119+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:11.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:11.119+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:11.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:11.119+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:11.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:11.427+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:11.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:11.427+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:11.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:11.427+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:11.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:11.427+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:11.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:11 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:11.427+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:12.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:12.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:12.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:12.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:12.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:12.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:11 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:11.786+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:12.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:12 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:12.116+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:12.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:12 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:12.116+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:12 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:12.116+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:12 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:12.116+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:12 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:12.116+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:11.889+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.842070+0000 front 2024-03-28T16:04:15.842091+0000 (oldest deadline 2024-03-28T16:04:38.742043+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:11.889+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:18.742580+0000 front 2024-03-28T16:04:18.742477+0000 (oldest deadline 2024-03-28T16:04:44.642686+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:11.889+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.444552+0000 front 2024-03-28T16:04:30.444637+0000 (oldest deadline 2024-03-28T16:04:53.944413+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:11.889+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.145970+0000 front 2024-03-28T16:04:39.145905+0000 (oldest deadline 2024-03-28T16:05:02.645805+0000) 2024-03-28T16:05:12.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8[55099]: 2024-03-28T16:05:11.889+0000 7fb66edd2700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.146748+0000 front 2024-03-28T16:04:46.146696+0000 (oldest deadline 2024-03-28T16:05:10.246767+0000) 2024-03-28T16:05:12.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:12 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:12.452+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:12.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:12 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:12.452+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:12.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:12 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:12.452+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:12.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:12 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:12.452+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:12.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:12 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:12.452+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:13.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:12 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:12.788+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:13.119 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:12 smithi120 podman[73354]: 2024-03-28 16:05:12.842471831 +0000 UTC m=+5.631042444 container died be0630eae8cca0893f9a5d439957a6b6810e4283a8d2e3673eb84ce50da856ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:13 smithi120 podman[73354]: 2024-03-28 16:05:13.226774431 +0000 UTC m=+6.015345046 container cleanup be0630eae8cca0893f9a5d439957a6b6810e4283a8d2e3673eb84ce50da856ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:13 smithi120 bash[73354]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:13.136+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:13.136+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:13.136+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:13.136+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:13.478 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:13 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:13.136+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:13.799 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:13 smithi120 podman[73371]: 2024-03-28 16:05:13.478136551 +0000 UTC m=+0.635527883 container remove be0630eae8cca0893f9a5d439957a6b6810e4283a8d2e3673eb84ce50da856ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-28T16:05:13.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:13 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:13.494+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:13.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:13 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:13.494+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:13.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:13 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:13.494+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:13.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:13 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:13.494+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:13.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:13 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:13.494+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:14.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:14.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:14.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:14.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:14.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:14.072 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:13 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:13.816+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:14.072 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:13 smithi120 podman[73464]: 2024-03-28 16:05:13.703159772 +0000 UTC m=+0.017679684 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:05:14.072 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:13 smithi120 podman[73464]: 2024-03-28 16:05:13.837339374 +0000 UTC m=+0.151859298 container create c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-28T16:05:14.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:14 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:14.166+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:14.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:14 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:14.166+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:14.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:14 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:14.166+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:14.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:14 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:14.166+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:14.340 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:14 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:14.166+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:14.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 podman[73464]: 2024-03-28 16:05:14.072978043 +0000 UTC m=+0.387497942 container init c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:05:14.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 podman[73464]: 2024-03-28 16:05:14.076140126 +0000 UTC m=+0.390660030 container start c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-28T16:05:14.340 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 podman[73464]: 2024-03-28 16:05:14.173118203 +0000 UTC m=+0.487638118 container attach c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-28T16:05:14.642 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 podman[73464]: 2024-03-28 16:05:14.359828108 +0000 UTC m=+0.674348025 container died c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T16:05:14.892 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 podman[73490]: 2024-03-28 16:05:14.876189005 +0000 UTC m=+0.511248136 container remove c173309efd3a7eb8d0004dea82afabf0a0c351a069cd965fd3e07e952be79a74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T16:05:14.892 INFO:journalctl@ceph.osd.8.smithi120.stdout:Mar 28 16:05:14 smithi120 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.8.service: Deactivated successfully. 2024-03-28T16:05:14.895 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.8.service' 2024-03-28T16:05:14.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:14 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:14.522+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:14.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:14 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:14.522+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:14.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:14 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:14.522+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:14.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:14 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:14.522+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:14.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:14 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:14.522+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:14.975 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:14.976 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-28T16:05:14.976 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-28T16:05:14.976 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.9 2024-03-28T16:05:15.157 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:15.156+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:15.157 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:15.156+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:15.157 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:15.156+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:15.158 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:15.156+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:15.158 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:15.156+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:15.290 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 systemd[1]: Stopping Ceph osd.9 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:15.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:14 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:14.842+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:15.562 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.290+0000 7fc379a18700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:15.563 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.290+0000 7fc379a18700 -1 osd.9 107 *** Got signal Terminated *** 2024-03-28T16:05:15.563 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.290+0000 7fc379a18700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:05:15.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.562+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:15.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.562+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:15.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.562+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:15.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.562+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:15.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:15 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:15.562+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:16.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:15 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:15.831+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:16.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:16 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:16.109+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:16.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:16 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:16.109+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:16.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:16 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:16.109+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:16.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:16 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:16.109+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:16.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:16 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:16.109+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:16.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:16 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:16.582+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:16.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:16 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:16.582+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:16.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:16 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:16.582+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:16.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:16 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:16.582+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:16.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:16 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:16.582+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:17.095 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:16 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:16.834+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:17.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:17 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:17.077+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:17.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:17 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:17.077+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:17.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:17 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:17.077+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:17.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:17 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:17.077+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:17.339 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:17 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:17.077+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:17.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:17 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:17.589+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:17.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:17 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:17.589+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:17.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:17 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:17.589+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:17.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:17 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:17.589+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:17.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:17 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:17.589+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:18.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:18.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:18.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:18.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:18.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:18.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:17 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:17.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:18.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:18 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:18.122+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:18.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:18 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:18.122+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:18.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:18 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:18.122+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:18.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:18 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:18.122+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:18.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:18 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:18.122+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:18.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:18 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:18.606+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:18.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:18 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:18.606+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:18.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:18 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:18.606+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:18.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:18 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:18.606+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:18.921 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:18 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:18.606+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:19.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:19.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:19.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:19.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:19.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:19.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:19.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:18 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:18.883+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:19.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:19.143+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:19.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:19 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:19.574+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:15.284056+0000 front 2024-03-28T16:04:15.284100+0000 (oldest deadline 2024-03-28T16:04:38.184045+0000) 2024-03-28T16:05:19.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:19 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:19.574+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.885531+0000 front 2024-03-28T16:04:22.885514+0000 (oldest deadline 2024-03-28T16:04:45.785443+0000) 2024-03-28T16:05:19.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:19 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:19.574+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:30.287683+0000 front 2024-03-28T16:04:30.287659+0000 (oldest deadline 2024-03-28T16:04:54.987815+0000) 2024-03-28T16:05:19.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:19 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:19.574+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.589588+0000 front 2024-03-28T16:04:39.589690+0000 (oldest deadline 2024-03-28T16:05:04.289533+0000) 2024-03-28T16:05:19.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:19 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9[67037]: 2024-03-28T16:05:19.574+0000 7fc37600c700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:45.990803+0000 front 2024-03-28T16:04:45.990782+0000 (oldest deadline 2024-03-28T16:05:08.890718+0000) 2024-03-28T16:05:20.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:20.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:20.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:20.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:20.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:20.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:20.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:19 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:19.899+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:20.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:20.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:20.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:20.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:20.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:20.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:20 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:20.167+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:21.064 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:20 smithi023 podman[93081]: 2024-03-28 16:05:20.759946061 +0000 UTC m=+5.590118442 container died 0a5e6a3ed83da567b678f2fd2c1b4243b0e794d4782d05a7a13bcd863ac8dca7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-28T16:05:21.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:21.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:21.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:21.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:21.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:21.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:21.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:20 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:20.895+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:21.420 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93081]: 2024-03-28 16:05:21.064720213 +0000 UTC m=+5.894892600 container cleanup 0a5e6a3ed83da567b678f2fd2c1b4243b0e794d4782d05a7a13bcd863ac8dca7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-28T16:05:21.420 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 bash[93081]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9 2024-03-28T16:05:21.420 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93099]: 2024-03-28 16:05:21.282609027 +0000 UTC m=+0.517801750 container remove 0a5e6a3ed83da567b678f2fd2c1b4243b0e794d4782d05a7a13bcd863ac8dca7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-28T16:05:21.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:21.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:21.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:21.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:21.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:21.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:21 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:21.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:21.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93189]: 2024-03-28 16:05:21.514165203 +0000 UTC m=+0.017895119 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:05:21.920 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93189]: 2024-03-28 16:05:21.741599366 +0000 UTC m=+0.245329269 container create 73a8dfcc75e736d42524d8ffa4fcc1735374cce55050e82bc2ebd1aafea765c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-28T16:05:22.272 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93189]: 2024-03-28 16:05:21.969513106 +0000 UTC m=+0.473243020 container init 73a8dfcc75e736d42524d8ffa4fcc1735374cce55050e82bc2ebd1aafea765c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T16:05:22.272 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:21 smithi023 podman[93189]: 2024-03-28 16:05:21.972672126 +0000 UTC m=+0.476402031 container start 73a8dfcc75e736d42524d8ffa4fcc1735374cce55050e82bc2ebd1aafea765c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-28T16:05:22.273 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:22 smithi023 podman[93189]: 2024-03-28 16:05:22.052290443 +0000 UTC m=+0.556020357 container attach 73a8dfcc75e736d42524d8ffa4fcc1735374cce55050e82bc2ebd1aafea765c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T16:05:22.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:22.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:22.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:22.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:22.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:22.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:22.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:21 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:21.892+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:22.553 INFO:journalctl@ceph.osd.9.smithi023.stdout:Mar 28 16:05:22 smithi023 podman[93189]: 2024-03-28 16:05:22.272806605 +0000 UTC m=+0.776536510 container died 73a8dfcc75e736d42524d8ffa4fcc1735374cce55050e82bc2ebd1aafea765c0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-9-deactivate, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-28T16:05:22.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:22.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:22.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:22.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:22.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:22.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:22 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:22.162+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:22.899 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.9.service' 2024-03-28T16:05:22.939 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:22.939 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-28T16:05:22.940 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-28T16:05:22.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.10 2024-03-28T16:05:23.233 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.851+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:23.233 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:23.233 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:22.852+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:22 smithi117 systemd[1]: Stopping Ceph osd.10 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.231+0000 7fee77ffa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.231+0000 7fee77ffa700 -1 osd.10 107 *** Got signal Terminated *** 2024-03-28T16:05:23.234 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.231+0000 7fee77ffa700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:05:23.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:23.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:23.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:23.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:23.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:23.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:23.168+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:24.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:24.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:24.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:24.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:24.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:24.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:24.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:23 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:23.885+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:24.490 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:24.491 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:24.491 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:24.491 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:24.491 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:24.491 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:24 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:24.211+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:25.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:25.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:25.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:25.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:25.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:25.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:25.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:24 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:24.907+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:25.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:25.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:25.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:25.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:25.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:25.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:25 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:25.225+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:26.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:26.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:26.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:26.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:26.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:26.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:26.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:25 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:25.896+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:26.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:26.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:26.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:26.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:26.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:26.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:26.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:26 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:26.188+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:27.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:27.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:26 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:26.848+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:27.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:27.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:27.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:27.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:27.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:27.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:27.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:27 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:27.174+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:28.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:05.368615+0000 front 2024-03-28T16:04:05.368636+0000 (oldest deadline 2024-03-28T16:04:28.868747+0000) 2024-03-28T16:05:28.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.670138+0000 front 2024-03-28T16:04:14.670288+0000 (oldest deadline 2024-03-28T16:04:40.570292+0000) 2024-03-28T16:05:28.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:20.570915+0000 front 2024-03-28T16:04:20.570858+0000 (oldest deadline 2024-03-28T16:04:49.271679+0000) 2024-03-28T16:05:28.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6814 osd.3 since back 2024-03-28T16:04:29.272198+0000 front 2024-03-28T16:04:29.272173+0000 (oldest deadline 2024-03-28T16:04:52.772159+0000) 2024-03-28T16:05:28.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:39.173603+0000 front 2024-03-28T16:04:39.173559+0000 (oldest deadline 2024-03-28T16:05:00.873857+0000) 2024-03-28T16:05:28.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:46.775260+0000 front 2024-03-28T16:04:46.775124+0000 (oldest deadline 2024-03-28T16:05:09.674751+0000) 2024-03-28T16:05:28.322 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:27 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10[60755]: 2024-03-28T16:05:27.857+0000 7fee705e6700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:52.575822+0000 front 2024-03-28T16:04:52.575865+0000 (oldest deadline 2024-03-28T16:05:17.875668+0000) 2024-03-28T16:05:28.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:28.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:28.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:28.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:28.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:28.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:28.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:28 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:28.189+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:29.071 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:28 smithi117 podman[72609]: 2024-03-28 16:05:28.764253534 +0000 UTC m=+5.630105898 container died 1e78d50206af2910073632f52ac2cd07c6d8b813b4b4916f926510e9fdf4ca16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-28T16:05:29.571 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72609]: 2024-03-28 16:05:29.082263443 +0000 UTC m=+5.948115861 container cleanup 1e78d50206af2910073632f52ac2cd07c6d8b813b4b4916f926510e9fdf4ca16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T16:05:29.571 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 bash[72609]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10 2024-03-28T16:05:29.571 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72626]: 2024-03-28 16:05:29.283386416 +0000 UTC m=+0.514548846 container remove 1e78d50206af2910073632f52ac2cd07c6d8b813b4b4916f926510e9fdf4ca16 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:29.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:29.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:29 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:29.205+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:29.977 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72720]: 2024-03-28 16:05:29.684227343 +0000 UTC m=+0.101844742 container create 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-28T16:05:29.977 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72720]: 2024-03-28 16:05:29.600206551 +0000 UTC m=+0.017823947 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:05:29.978 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72720]: 2024-03-28 16:05:29.902997755 +0000 UTC m=+0.320615134 container init 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-28T16:05:29.978 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72720]: 2024-03-28 16:05:29.906138077 +0000 UTC m=+0.323755464 container start 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:29.978 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:29 smithi117 podman[72720]: 2024-03-28 16:05:29.977852703 +0000 UTC m=+0.395470075 container attach 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-28T16:05:30.321 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:30 smithi117 podman[72720]: 2024-03-28 16:05:30.196135498 +0000 UTC m=+0.613752879 container died 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-28T16:05:30.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:30.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:30.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:30.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:30.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:30.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:30.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:30.187+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:30.709 INFO:journalctl@ceph.osd.10.smithi117.stdout:Mar 28 16:05:30 smithi117 podman[72747]: 2024-03-28 16:05:30.706209822 +0000 UTC m=+0.504512849 container remove 2abc8b78a1631e9a0009322bb9380eb345fab0039bf5d5c69bd4ab8cbb6c0c20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-10-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-28T16:05:30.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.10.service' 2024-03-28T16:05:30.777 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:30.777 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-28T16:05:30.777 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-28T16:05:30.778 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.11 2024-03-28T16:05:31.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:30 smithi120 systemd[1]: Stopping Ceph osd.11 for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:31.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.052+0000 7f55fc17b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:31.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.052+0000 7f55fc17b700 -1 osd.11 107 *** Got signal Terminated *** 2024-03-28T16:05:31.090 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.052+0000 7f55fc17b700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-28T16:05:31.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:31.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:31.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:31.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:31.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:31.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:31.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:31 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:31.152+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:32.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:32.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:32 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:32.144+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:33.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:33.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:33.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:33.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:33.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:33.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:33.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:33 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:33.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:34.545 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:34 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:34.140+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6820 osd.8 since back 2024-03-28T16:05:09.862134+0000 front 2024-03-28T16:05:09.862160+0000 (oldest deadline 2024-03-28T16:05:33.962162+0000) 2024-03-28T16:05:35.589 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6806 osd.0 since back 2024-03-28T16:04:06.849680+0000 front 2024-03-28T16:04:06.849784+0000 (oldest deadline 2024-03-28T16:04:32.749724+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-03-28T16:04:14.451067+0000 front 2024-03-28T16:04:14.451258+0000 (oldest deadline 2024-03-28T16:04:39.751014+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6804 osd.2 since back 2024-03-28T16:04:22.552646+0000 front 2024-03-28T16:04:22.552511+0000 (oldest deadline 2024-03-28T16:04:45.452690+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6812 osd.4 since back 2024-03-28T16:04:38.855147+0000 front 2024-03-28T16:04:38.855169+0000 (oldest deadline 2024-03-28T16:05:02.855565+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6812 osd.5 since back 2024-03-28T16:04:44.456969+0000 front 2024-03-28T16:04:44.457050+0000 (oldest deadline 2024-03-28T16:05:10.257543+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.23:6822 osd.6 since back 2024-03-28T16:04:54.358430+0000 front 2024-03-28T16:04:54.358354+0000 (oldest deadline 2024-03-28T16:05:18.458322+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.117:6820 osd.7 since back 2024-03-28T16:05:01.160732+0000 front 2024-03-28T16:05:01.160702+0000 (oldest deadline 2024-03-28T16:05:25.260626+0000) 2024-03-28T16:05:35.590 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:35 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11[61883]: 2024-03-28T16:05:35.133+0000 7f55f876f700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.120:6820 osd.8 since back 2024-03-28T16:05:09.862134+0000 front 2024-03-28T16:05:09.862160+0000 (oldest deadline 2024-03-28T16:05:33.962162+0000) 2024-03-28T16:05:36.803 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:36 smithi120 podman[73580]: 2024-03-28 16:05:36.529429099 +0000 UTC m=+5.552934191 container died 2b62c9d7592b8f87a5c054ff41b69edf8dde2aa843e4901c0add01625dc4af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-28T16:05:37.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:36 smithi120 podman[73580]: 2024-03-28 16:05:36.902895906 +0000 UTC m=+5.926400981 container cleanup 2b62c9d7592b8f87a5c054ff41b69edf8dde2aa843e4901c0add01625dc4af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:05:37.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:36 smithi120 bash[73580]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11 2024-03-28T16:05:37.520 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73598]: 2024-03-28 16:05:37.195538449 +0000 UTC m=+0.664379328 container remove 2b62c9d7592b8f87a5c054ff41b69edf8dde2aa843e4901c0add01625dc4af4b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:37.831 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73691]: 2024-03-28 16:05:37.423368139 +0000 UTC m=+0.017401816 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-28T16:05:37.831 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73691]: 2024-03-28 16:05:37.563027597 +0000 UTC m=+0.157061300 container create 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:37.831 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73691]: 2024-03-28 16:05:37.756756503 +0000 UTC m=+0.350790168 container init 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:05:37.831 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73691]: 2024-03-28 16:05:37.760218903 +0000 UTC m=+0.354252562 container start 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-28T16:05:37.831 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:37 smithi120 podman[73691]: 2024-03-28 16:05:37.83161103 +0000 UTC m=+0.425644692 container attach 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:38.089 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:38 smithi120 podman[73691]: 2024-03-28 16:05:38.052860533 +0000 UTC m=+0.646894186 container died 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-28T16:05:38.600 INFO:journalctl@ceph.osd.11.smithi120.stdout:Mar 28 16:05:38 smithi120 podman[73718]: 2024-03-28 16:05:38.576786162 +0000 UTC m=+0.518249141 container remove 268acafdaef28a0875879caec59183b4a0a0be2eaeec9da5cc1794283ce65a59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-28T16:05:38.606 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@osd.11.service' 2024-03-28T16:05:38.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:38.648 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-28T16:05:38.648 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-28T16:05:38.648 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.a 2024-03-28T16:05:38.958 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:38 smithi023 systemd[1]: Stopping Ceph mds.a for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:39.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:38 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: 2024-03-28T16:05:38.957+0000 7fdef2ade700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:39.420 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:38 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a[73548]: 2024-03-28T16:05:38.957+0000 7fdef2ade700 -1 mds.a *** got signal Terminated *** 2024-03-28T16:05:42.670 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:42 smithi023 podman[93310]: 2024-03-28 16:05:42.372032994 +0000 UTC m=+3.520508180 container died 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:05:43.103 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:42 smithi023 podman[93310]: 2024-03-28 16:05:42.848833245 +0000 UTC m=+3.997308435 container cleanup 694630879cadd4feab29065843dcd54a9c4e0080dc911208cd4a0cd6b2c5babf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:05:43.104 INFO:journalctl@ceph.mds.a.smithi023.stdout:Mar 28 16:05:42 smithi023 bash[93310]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-a 2024-03-28T16:05:43.223 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.a.service' 2024-03-28T16:05:43.259 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:43.259 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-28T16:05:43.260 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-28T16:05:43.260 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.d 2024-03-28T16:05:43.420 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:43 smithi023 systemd[1]: Stopping Ceph mds.d for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:43.920 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:43 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d[72591]: 2024-03-28T16:05:43.582+0000 7f53c691c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:43.920 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:43 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d[72591]: 2024-03-28T16:05:43.582+0000 7f53c691c700 -1 mds.d *** got signal Terminated *** 2024-03-28T16:05:45.624 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:45 smithi023 podman[93430]: 2024-03-28 16:05:45.373869594 +0000 UTC m=+1.895113430 container died e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:05:45.920 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:45 smithi023 podman[93430]: 2024-03-28 16:05:45.706800033 +0000 UTC m=+2.228043869 container cleanup e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:05:45.920 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:45 smithi023 bash[93430]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d 2024-03-28T16:05:45.920 INFO:journalctl@ceph.mds.d.smithi023.stdout:Mar 28 16:05:45 smithi023 podman[93444]: 2024-03-28 16:05:45.874580345 +0000 UTC m=+0.496541340 container remove e7d17312026da32cfe424f2126f7c9115f5821e3b768e61401db18390e3d78e4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-28T16:05:46.094 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.d.service' 2024-03-28T16:05:46.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:46.128 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-28T16:05:46.128 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-28T16:05:46.128 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.g 2024-03-28T16:05:46.420 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:46 smithi023 systemd[1]: Stopping Ceph mds.g for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:46.920 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: 2024-03-28T16:05:46.454+0000 7fd607d6f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:46.920 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:46 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g[73186]: 2024-03-28T16:05:46.454+0000 7fd607d6f700 -1 mds.g *** got signal Terminated *** 2024-03-28T16:05:50.627 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:50 smithi023 podman[93552]: 2024-03-28 16:05:50.369786608 +0000 UTC m=+4.024516846 container died be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-28T16:05:50.902 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:50 smithi023 podman[93552]: 2024-03-28 16:05:50.701607975 +0000 UTC m=+4.356338204 container cleanup be382c12b7ee068d0da0f679c385445ba02c3395e9830acad21e93eef2b878f6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-28T16:05:50.902 INFO:journalctl@ceph.mds.g.smithi023.stdout:Mar 28 16:05:50 smithi023 bash[93552]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-g 2024-03-28T16:05:51.126 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.g.service' 2024-03-28T16:05:51.161 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:51.161 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-28T16:05:51.161 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-28T16:05:51.161 DEBUG:teuthology.orchestra.run.smithi023:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.j 2024-03-28T16:05:51.420 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:51 smithi023 systemd[1]: Stopping Ceph mds.j for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:51.920 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j[72245]: 2024-03-28T16:05:51.480+0000 7f4acfa71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:51.920 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:51 smithi023 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j[72245]: 2024-03-28T16:05:51.480+0000 7f4acfa71700 -1 mds.j *** got signal Terminated *** 2024-03-28T16:05:54.797 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:54 smithi023 podman[93672]: 2024-03-28 16:05:54.516359656 +0000 UTC m=+3.138746373 container died 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-28T16:05:55.106 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:54 smithi023 podman[93672]: 2024-03-28 16:05:54.871929044 +0000 UTC m=+3.494315771 container cleanup 851f4e4119431f47690450e67c631639b7b187c3e4fca67c6c773e17de618fad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-28T16:05:55.106 INFO:journalctl@ceph.mds.j.smithi023.stdout:Mar 28 16:05:54 smithi023 bash[93672]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-j 2024-03-28T16:05:55.249 DEBUG:teuthology.orchestra.run.smithi023:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.j.service' 2024-03-28T16:05:55.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:55.282 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-28T16:05:55.282 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-28T16:05:55.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.b 2024-03-28T16:05:55.571 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:55 smithi117 systemd[1]: Stopping Ceph mds.b for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:56.071 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T16:05:55.579+0000 7f5ec5304700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:56.071 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:55 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b[66323]: 2024-03-28T16:05:55.579+0000 7f5ec5304700 -1 mds.b *** got signal Terminated *** 2024-03-28T16:05:58.639 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:58 smithi117 podman[72839]: 2024-03-28 16:05:58.380361839 +0000 UTC m=+2.905838194 container died cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-28T16:05:58.997 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:58 smithi117 podman[72839]: 2024-03-28 16:05:58.737702354 +0000 UTC m=+3.263178701 container cleanup cbc055eb4cfa9634bf39ca6b142ccf5a61086de45b15b4fa4b46865d2e1e7edb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-28T16:05:58.997 INFO:journalctl@ceph.mds.b.smithi117.stdout:Mar 28 16:05:58 smithi117 bash[72839]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-b 2024-03-28T16:05:59.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.b.service' 2024-03-28T16:05:59.200 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:05:59.200 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-28T16:05:59.200 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-28T16:05:59.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.e 2024-03-28T16:05:59.516 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:05:59 smithi117 systemd[1]: Stopping Ceph mds.e for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:05:59.821 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:05:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: 2024-03-28T16:05:59.515+0000 7f657dce0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:05:59.821 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:05:59 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e[65697]: 2024-03-28T16:05:59.515+0000 7f657dce0700 -1 mds.e *** got signal Terminated *** 2024-03-28T16:06:02.634 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:06:02 smithi117 podman[72959]: 2024-03-28 16:06:02.371862147 +0000 UTC m=+2.944636962 container died 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-28T16:06:02.984 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:06:02 smithi117 podman[72959]: 2024-03-28 16:06:02.724298714 +0000 UTC m=+3.297073525 container cleanup 39baf81c7888bc47588e2efe0fdcacb09180350ceba23b8bc47f63b8be648417 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:06:02.984 INFO:journalctl@ceph.mds.e.smithi117.stdout:Mar 28 16:06:02 smithi117 bash[72959]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-e 2024-03-28T16:06:03.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.e.service' 2024-03-28T16:06:03.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:03.187 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-28T16:06:03.188 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-28T16:06:03.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.h 2024-03-28T16:06:03.571 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:03 smithi117 systemd[1]: Stopping Ceph mds.h for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:03.571 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: 2024-03-28T16:06:03.484+0000 7fe476075700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:03.571 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:03 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h[66008]: 2024-03-28T16:06:03.484+0000 7fe476075700 -1 mds.h *** got signal Terminated *** 2024-03-28T16:06:06.645 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:06 smithi117 podman[73092]: 2024-03-28 16:06:06.376227491 +0000 UTC m=+2.976770464 container died 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:06:06.929 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:06 smithi117 podman[73092]: 2024-03-28 16:06:06.743958128 +0000 UTC m=+3.344501095 container cleanup 210c59df0a6f3b5c924aad18561a9cd3db48b6184e3752cbb7ecea3f511548f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-28T16:06:06.929 INFO:journalctl@ceph.mds.h.smithi117.stdout:Mar 28 16:06:06 smithi117 bash[73092]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-h 2024-03-28T16:06:07.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.h.service' 2024-03-28T16:06:07.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:07.185 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-28T16:06:07.185 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-28T16:06:07.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.k 2024-03-28T16:06:07.506 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:07 smithi117 systemd[1]: Stopping Ceph mds.k for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:07.821 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: 2024-03-28T16:06:07.505+0000 7f53fb86a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:07.821 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:07 smithi117 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k[65376]: 2024-03-28T16:06:07.505+0000 7f53fb86a700 -1 mds.k *** got signal Terminated *** 2024-03-28T16:06:10.690 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:10 smithi117 podman[73213]: 2024-03-28 16:06:10.386286815 +0000 UTC m=+2.987984865 container died ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-28T16:06:10.951 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:10 smithi117 podman[73213]: 2024-03-28 16:06:10.764244113 +0000 UTC m=+3.365942182 container cleanup ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-28T16:06:10.951 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:10 smithi117 bash[73213]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k 2024-03-28T16:06:11.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.k.service' 2024-03-28T16:06:11.214 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:11 smithi117 podman[73227]: 2024-03-28 16:06:11.040089652 +0000 UTC m=+0.649808094 container remove ee4a1c5f88520e8a0f5a17bd8c9e9f4f1c61072cdc2fced1041a1c4dd3dbe27e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-28T16:06:11.214 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:11 smithi117 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.k.service: Deactivated successfully. 2024-03-28T16:06:11.214 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:11 smithi117 systemd[1]: Stopped Ceph mds.k for a96af96e-ed19-11ee-95d0-87774f69a715. 2024-03-28T16:06:11.214 INFO:journalctl@ceph.mds.k.smithi117.stdout:Mar 28 16:06:11 smithi117 systemd[1]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.k.service: Consumed 1.685s CPU time. 2024-03-28T16:06:11.225 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:11.225 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-28T16:06:11.225 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-28T16:06:11.225 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.c 2024-03-28T16:06:11.510 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:11 smithi120 systemd[1]: Stopping Ceph mds.c for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:11.779 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: 2024-03-28T16:06:11.509+0000 7f14f7ac3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:11.779 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:11 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c[66784]: 2024-03-28T16:06:11.509+0000 7f14f7ac3700 -1 mds.c *** got signal Terminated *** 2024-03-28T16:06:14.839 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:14 smithi120 podman[73799]: 2024-03-28 16:06:14.372747331 +0000 UTC m=+2.956629999 container died 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-28T16:06:15.318 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:15 smithi120 podman[73799]: 2024-03-28 16:06:15.050593097 +0000 UTC m=+3.634475761 container cleanup 07359ac8615211a79a09d36297c8ab142d59fe3d15bea0b0735e41a8ff5b060e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-28T16:06:15.318 INFO:journalctl@ceph.mds.c.smithi120.stdout:Mar 28 16:06:15 smithi120 bash[73799]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-c 2024-03-28T16:06:15.489 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.c.service' 2024-03-28T16:06:15.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:15.527 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-28T16:06:15.527 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-28T16:06:15.527 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.f 2024-03-28T16:06:16.089 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:15 smithi120 systemd[1]: Stopping Ceph mds.f for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:16.089 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: 2024-03-28T16:06:15.812+0000 7f21e24e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:16.090 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:15 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f[66479]: 2024-03-28T16:06:15.812+0000 7f21e24e5700 -1 mds.f *** got signal Terminated *** 2024-03-28T16:06:18.630 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:18 smithi120 podman[73933]: 2024-03-28 16:06:18.37472595 +0000 UTC m=+2.638893897 container died 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-28T16:06:19.014 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:18 smithi120 podman[73933]: 2024-03-28 16:06:18.737675848 +0000 UTC m=+3.001843755 container cleanup 64a4d56bb72df69a647063b925d500717fdd2b74e14ae215e4b1fcb26208cedc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-28T16:06:19.014 INFO:journalctl@ceph.mds.f.smithi120.stdout:Mar 28 16:06:18 smithi120 bash[73933]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-f 2024-03-28T16:06:19.112 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.f.service' 2024-03-28T16:06:19.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:19.148 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-28T16:06:19.148 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-28T16:06:19.148 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.i 2024-03-28T16:06:19.339 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:19 smithi120 systemd[1]: Stopping Ceph mds.i for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:19.841 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i[67098]: 2024-03-28T16:06:19.457+0000 7f819fc48700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:19.841 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:19 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i[67098]: 2024-03-28T16:06:19.457+0000 7f819fc48700 -1 mds.i *** got signal Terminated *** 2024-03-28T16:06:22.589 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:22 smithi120 podman[74055]: 2024-03-28 16:06:22.368994609 +0000 UTC m=+3.005318630 container died d7784bfc505410aec4796e19209c7d24a6890bdaaa6b3a2377f5652ebd8e72ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-28T16:06:23.084 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:22 smithi120 podman[74055]: 2024-03-28 16:06:22.849370901 +0000 UTC m=+3.485694856 container cleanup d7784bfc505410aec4796e19209c7d24a6890bdaaa6b3a2377f5652ebd8e72ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-28T16:06:23.084 INFO:journalctl@ceph.mds.i.smithi120.stdout:Mar 28 16:06:22 smithi120 bash[74055]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-i 2024-03-28T16:06:23.193 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.i.service' 2024-03-28T16:06:23.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:23.228 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-28T16:06:23.228 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-28T16:06:23.228 DEBUG:teuthology.orchestra.run.smithi120:> sudo systemctl stop ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.l 2024-03-28T16:06:23.339 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:23 smithi120 systemd[1]: Stopping Ceph mds.l for a96af96e-ed19-11ee-95d0-87774f69a715... 2024-03-28T16:06:23.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: 2024-03-28T16:06:23.527+0000 7fe3879ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-28T16:06:23.839 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:23 smithi120 ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l[66175]: 2024-03-28T16:06:23.527+0000 7fe3879ae700 -1 mds.l *** got signal Terminated *** 2024-03-28T16:06:26.678 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:26 smithi120 podman[74176]: 2024-03-28 16:06:26.372820331 +0000 UTC m=+2.931214585 container died 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-28T16:06:27.030 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:26 smithi120 podman[74176]: 2024-03-28 16:06:26.763176079 +0000 UTC m=+3.321570323 container cleanup 7a0c590f3371f552802352f3f20dbaf720045140b554a883ca39b28a38e9752c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-28T16:06:27.031 INFO:journalctl@ceph.mds.l.smithi120.stdout:Mar 28 16:06:26 smithi120 bash[74176]: ceph-a96af96e-ed19-11ee-95d0-87774f69a715-mds-l 2024-03-28T16:06:27.181 DEBUG:teuthology.orchestra.run.smithi120:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a96af96e-ed19-11ee-95d0-87774f69a715@mds.l.service' 2024-03-28T16:06:27.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-28T16:06:27.214 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-28T16:06:27.214 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:06:27.410 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:29.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:06:30.056 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:32.619 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-28T16:06:32.807 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:35.175 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:06:35.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:06:35.245 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-28T16:06:35.273 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-28T16:06:35.276 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi023/crash 2024-03-28T16:06:35.277 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:06:35.326 INFO:teuthology.orchestra.run.smithi023.stderr:tar: /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:06:35.326 INFO:teuthology.orchestra.run.smithi023.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:06:35.328 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi117/crash 2024-03-28T16:06:35.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:06:35.367 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:06:35.367 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:06:35.369 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi120/crash 2024-03-28T16:06:35.370 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash -- . 2024-03-28T16:06:35.404 INFO:teuthology.orchestra.run.smithi120.stderr:tar: /var/lib/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-28T16:06:35.405 INFO:teuthology.orchestra.run.smithi120.stderr:tar: Error is not recoverable: exiting now 2024-03-28T16:06:35.406 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-28T16:06:35.406 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T16:06:35.445 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-28T15:52:37.274994+0000 mon.a (mon.0) 1144 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-28T16:06:35.446 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-28T16:06:35.446 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[SEC\]' /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T16:06:35.521 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[ERR\]' /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T16:06:35.595 DEBUG:teuthology.orchestra.run.smithi023:> sudo egrep '\[WRN\]' /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | head -n 1 2024-03-28T16:06:35.668 INFO:teuthology.orchestra.run.smithi023.stdout:2024-03-28T15:52:37.274994+0000 mon.a (mon.0) 1144 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-28T16:06:35.668 INFO:tasks.cephadm:Compressing logs... 2024-03-28T16:06:35.668 DEBUG:teuthology.orchestra.run.smithi023:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:06:35.710 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:06:35.713 DEBUG:teuthology.orchestra.run.smithi120:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T16:06:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:06:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:: No such file or directory 2024-03-28T16:06:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-28T16:06:35.735 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-28T16:06:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:06:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-28T16:06:35.736 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-28T16:06:35.737 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:: No such file or directory 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.l.log: /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-28T16:06:35.742 INFO:teuthology.orchestra.run.smithi120.stderr: 92.4%/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-28T16:06:35.743 INFO:teuthology.orchestra.run.smithi120.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:06:35.743 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-28T16:06:35.743 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:06:35.752 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.4% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:06:35.752 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:06:35.752 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:06:35.752 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-28T16:06:35.752 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log: 87.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:06:35.753 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-28T16:06:35.753 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.2.log: 91.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-28T16:06:35.753 INFO:teuthology.orchestra.run.smithi120.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:06:35.754 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:: No such file or directory 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.4% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:06:35.755 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.3.log: /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 90.3% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr: --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr: 84.2%/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log: -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr: 87.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-28T16:06:35.756 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-28T16:06:35.760 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-28T16:06:35.762 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-28T16:06:35.763 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.d.log: 91.0% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:06:35.763 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-28T16:06:35.768 INFO:teuthology.orchestra.run.smithi023.stderr: 94.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-28T16:06:35.768 INFO:teuthology.orchestra.run.smithi023.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-28T16:06:35.774 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.y.log: 91.8%gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log 2024-03-28T16:06:35.774 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-28T16:06:35.781 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-28T16:06:35.781 INFO:teuthology.orchestra.run.smithi117.stderr: -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-28T16:06:35.783 INFO:teuthology.orchestra.run.smithi023.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-28T16:06:35.792 INFO:teuthology.orchestra.run.smithi120.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:06:35.794 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log: 87.6%gzip -5 --verbose -- /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-28T16:06:35.797 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log: -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-28T16:06:35.800 INFO:teuthology.orchestra.run.smithi120.stderr: 93.1% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-28T16:06:35.803 INFO:teuthology.orchestra.run.smithi023.stderr: 94.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-28T16:06:35.803 INFO:teuthology.orchestra.run.smithi120.stderr: 94.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-28T16:06:35.804 INFO:teuthology.orchestra.run.smithi117.stderr: 93.2% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-28T16:06:35.805 INFO:teuthology.orchestra.run.smithi120.stderr: 93.3% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-28T16:06:35.814 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-28T16:06:35.816 INFO:teuthology.orchestra.run.smithi117.stderr: 90.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-28T16:06:35.826 INFO:teuthology.orchestra.run.smithi117.stderr: 94.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-28T16:06:35.830 INFO:teuthology.orchestra.run.smithi120.stderr: 93.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-28T16:06:35.837 INFO:teuthology.orchestra.run.smithi023.stderr: 93.0% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-28T16:06:35.989 INFO:teuthology.orchestra.run.smithi023.stderr: 89.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-28T16:06:36.037 INFO:teuthology.orchestra.run.smithi117.stderr: 93.1% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-28T16:06:36.168 INFO:teuthology.orchestra.run.smithi117.stderr: 92.0% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-28T16:06:36.258 INFO:teuthology.orchestra.run.smithi120.stderr: 91.7% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-28T16:06:36.957 INFO:teuthology.orchestra.run.smithi023.stderr: 90.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-28T16:06:37.754 INFO:teuthology.orchestra.run.smithi120.stderr: 92.2% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-28T16:06:38.377 INFO:teuthology.orchestra.run.smithi023.stderr: 91.9% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-28T16:06:38.520 INFO:teuthology.orchestra.run.smithi117.stderr: 92.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-28T16:06:38.625 INFO:teuthology.orchestra.run.smithi117.stderr: 92.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-28T16:06:38.957 INFO:teuthology.orchestra.run.smithi120.stderr: 91.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-28T16:06:39.292 INFO:teuthology.orchestra.run.smithi117.stderr: 92.3% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-28T16:06:39.533 INFO:teuthology.orchestra.run.smithi023.stderr: 92.2% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-28T16:06:39.734 INFO:teuthology.orchestra.run.smithi120.stderr: 91.5% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-28T16:06:39.872 INFO:teuthology.orchestra.run.smithi117.stderr: 91.6% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-28T16:06:39.874 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-28T16:06:39.874 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m4.147s 2024-03-28T16:06:39.874 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m13.741s 2024-03-28T16:06:39.874 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.474s 2024-03-28T16:06:40.359 INFO:teuthology.orchestra.run.smithi023.stderr: 92.0% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-28T16:06:40.373 INFO:teuthology.orchestra.run.smithi120.stderr: 92.1% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-28T16:06:40.375 INFO:teuthology.orchestra.run.smithi120.stderr: 2024-03-28T16:06:40.375 INFO:teuthology.orchestra.run.smithi120.stderr:real 0m4.648s 2024-03-28T16:06:40.375 INFO:teuthology.orchestra.run.smithi120.stderr:user 0m14.081s 2024-03-28T16:06:40.375 INFO:teuthology.orchestra.run.smithi120.stderr:sys 0m0.498s 2024-03-28T16:06:41.393 INFO:teuthology.orchestra.run.smithi023.stderr: 91.4% -- replaced with /var/log/ceph/a96af96e-ed19-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-28T16:06:41.395 INFO:teuthology.orchestra.run.smithi023.stderr: 2024-03-28T16:06:41.396 INFO:teuthology.orchestra.run.smithi023.stderr:real 0m5.673s 2024-03-28T16:06:41.396 INFO:teuthology.orchestra.run.smithi023.stderr:user 0m17.813s 2024-03-28T16:06:41.396 INFO:teuthology.orchestra.run.smithi023.stderr:sys 0m0.534s 2024-03-28T16:06:41.396 INFO:tasks.cephadm:Archiving logs... 2024-03-28T16:06:41.396 DEBUG:teuthology.misc:Transferring archived files from smithi023:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi023/log 2024-03-28T16:06:41.397 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:06:44.258 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi117/log 2024-03-28T16:06:44.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:06:46.610 DEBUG:teuthology.misc:Transferring archived files from smithi120:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi120/log 2024-03-28T16:06:46.611 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T16:06:49.109 INFO:tasks.cephadm:Removing cluster... 2024-03-28T16:06:49.109 DEBUG:teuthology.orchestra.run.smithi023:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force 2024-03-28T16:06:49.308 INFO:teuthology.orchestra.run.smithi023.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:49.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force 2024-03-28T16:06:49.774 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:50.047 DEBUG:teuthology.orchestra.run.smithi120:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a96af96e-ed19-11ee-95d0-87774f69a715 --force 2024-03-28T16:06:50.240 INFO:teuthology.orchestra.run.smithi120.stdout:Deleting cluster with fsid: a96af96e-ed19-11ee-95d0-87774f69a715 2024-03-28T16:06:50.635 INFO:tasks.cephadm:Removing cephadm ... 2024-03-28T16:06:50.636 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:06:50.654 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:06:50.669 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-28T16:06:50.684 INFO:tasks.cephadm:Teardown complete 2024-03-28T16:06:50.684 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T16:06:50.697 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T16:06:50.697 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:06:50.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:06:50.712 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T16:06:50.815 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T16:06:50.825 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T16:06:50.825 DEBUG:teuthology.orchestra.run.smithi023:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:06:50.828 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:06:50.830 DEBUG:teuthology.orchestra.run.smithi120:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T16:06:50.841 INFO:teuthology.orchestra.run.smithi023.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:06:50.841 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:06:50.844 INFO:teuthology.orchestra.run.smithi120.stderr:bash: line 1: ntpq: command not found 2024-03-28T16:06:50.845 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:06:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-28T16:06:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +399us[ +405us] +/- 72ms 2024-03-28T16:06:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -189us[ -184us] +/- 55ms 2024-03-28T16:06:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -112us[ -112us] +/- 65ms 2024-03-28T16:06:50.846 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:06:50.847 INFO:teuthology.orchestra.run.smithi023.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:06:50.847 INFO:teuthology.orchestra.run.smithi023.stdout:=============================================================================== 2024-03-28T16:06:50.848 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 16 +404us[ +404us] +/- 72ms 2024-03-28T16:06:50.848 INFO:teuthology.orchestra.run.smithi023.stdout:^* hv02.front.sepia.ceph.com 3 6 377 16 -181us[ -163us] +/- 55ms 2024-03-28T16:06:50.848 INFO:teuthology.orchestra.run.smithi023.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -102us[ -85us] +/- 65ms 2024-03-28T16:06:50.848 INFO:teuthology.orchestra.run.smithi023.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:=============================================================================== 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 18 +375us[ +383us] +/- 72ms 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -178us[ -170us] +/- 55ms 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -120us[ -120us] +/- 65ms 2024-03-28T16:06:50.849 INFO:teuthology.orchestra.run.smithi120.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T16:06:50.849 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T16:06:50.859 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T16:06:50.859 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T16:06:50.877 DEBUG:teuthology.orchestra.run.smithi023:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:06:50.973 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:06:51.043 DEBUG:teuthology.orchestra.run.smithi120:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T16:06:51.113 DEBUG:teuthology.orchestra.run.smithi023:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:06:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:51.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:06:51.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:51.175 DEBUG:teuthology.orchestra.run.smithi120:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T16:06:51.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:51.204 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T16:06:51.216 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T16:06:51.304 INFO:teuthology.task.internal:Duration was 1940.926159 seconds 2024-03-28T16:06:51.304 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T16:06:51.320 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T16:06:51.320 DEBUG:teuthology.orchestra.run.smithi023:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:06:51.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:06:51.324 DEBUG:teuthology.orchestra.run.smithi120:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T16:06:51.364 INFO:teuthology.orchestra.run.smithi023.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:06:51.364 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:06:51.367 INFO:teuthology.orchestra.run.smithi120.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T16:06:51.866 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T16:06:51.866 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi023.front.sepia.ceph.com 2024-03-28T16:06:51.866 DEBUG:teuthology.orchestra.run.smithi023:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:06:51.887 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-28T16:06:51.887 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:06:51.918 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi120.front.sepia.ceph.com 2024-03-28T16:06:51.918 DEBUG:teuthology.orchestra.run.smithi120:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T16:06:51.947 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T16:06:51.947 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:06:51.949 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:06:51.960 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T16:06:52.015 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T16:06:52.016 DEBUG:teuthology.orchestra.run.smithi023:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:06:52.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:06:52.028 DEBUG:teuthology.orchestra.run.smithi120:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T16:06:52.419 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T16:06:52.431 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T16:06:52.431 DEBUG:teuthology.orchestra.run.smithi023:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:06:52.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:06:52.485 DEBUG:teuthology.orchestra.run.smithi120:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T16:06:52.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T16:06:52.524 DEBUG:teuthology.orchestra.run.smithi023:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.529 DEBUG:teuthology.orchestra.run.smithi120:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.549 INFO:teuthology.orchestra.run.smithi023.stdout:kernel.core_pattern = core 2024-03-28T16:06:52.552 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-28T16:06:52.578 INFO:teuthology.orchestra.run.smithi120.stdout:kernel.core_pattern = core 2024-03-28T16:06:52.601 DEBUG:teuthology.orchestra.run.smithi023:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:52.617 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:52.631 DEBUG:teuthology.orchestra.run.smithi120:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T16:06:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T16:06:52.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T16:06:52.666 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T16:06:52.667 DEBUG:teuthology.misc:Transferring archived files from smithi023:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi023 2024-03-28T16:06:52.667 DEBUG:teuthology.orchestra.run.smithi023:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:06:53.007 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi117 2024-03-28T16:06:53.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:06:53.053 DEBUG:teuthology.misc:Transferring archived files from smithi120:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315/remote/smithi120 2024-03-28T16:06:53.053 DEBUG:teuthology.orchestra.run.smithi120:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T16:06:53.093 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T16:06:53.093 DEBUG:teuthology.orchestra.run.smithi023:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:06:53.096 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:06:53.098 DEBUG:teuthology.orchestra.run.smithi120:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T16:06:53.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T16:06:53.161 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T16:06:53.161 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T16:06:53.172 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T16:06:53.172 DEBUG:teuthology.orchestra.run.smithi023:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:06:53.178 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:06:53.180 DEBUG:teuthology.orchestra.run.smithi120:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T16:06:53.187 INFO:teuthology.orchestra.run.smithi023.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:06 /home/ubuntu/cephtest 2024-03-28T16:06:53.192 INFO:teuthology.orchestra.run.smithi117.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:06 /home/ubuntu/cephtest 2024-03-28T16:06:53.196 INFO:teuthology.orchestra.run.smithi120.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 16:06 /home/ubuntu/cephtest 2024-03-28T16:06:53.197 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T16:06:53.209 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T16:06:53.232 INFO:teuthology.nuke:Checking targets against current locks 2024-03-28T16:06:53.256 DEBUG:teuthology.nuke:shortname: smithi023 2024-03-28T16:06:53.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:06:53.281 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi023.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.929993', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJKe2XaK9HWq33BrKGOTSxvkkDwiSln13+HoUlnMOxnPKisYU0HCrPUT39s6k1s7W7dSOC8TsF7ahA39Ny69wxA='} 2024-03-28T16:06:53.285 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-28T16:06:53.285 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:06:53.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.933697', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOKIwvfVbhEEto+ufiP/WeBcqdpEAiC5Npdu0Pn+eVJafT8N5Qx11/74qcv6FomniomKYvzxZPdRZHMd+MPyIWI='} 2024-03-28T16:06:53.313 DEBUG:teuthology.nuke:shortname: smithi120 2024-03-28T16:06:53.314 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T16:06:53.336 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi120.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-27_03:45:23-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7625315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 15:24:16.931971', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIyLah9Ww/6g3VLLFTwgmiScvB4FsQ+BcrTIkRx01rkcfEwkPFuIU9wGCETnGqY4NyaqNN/SjlV8/J1YiZ2+Z+Y='} 2024-03-28T16:06:53.344 INFO:teuthology.orchestra.console.smithi023:Power off 2024-03-28T16:06:53.344 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:06:53.367 DEBUG:teuthology.orchestra.console.smithi023:power off output: Chassis Power Control: Down/Off 2024-03-28T16:06:53.367 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:53.385 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-28T16:06:53.385 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:06:53.398 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-28T16:06:53.408 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-28T16:06:53.408 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:53.425 INFO:teuthology.orchestra.console.smithi120:Power off 2024-03-28T16:06:53.425 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-28T16:06:53.437 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-28T16:06:53.448 DEBUG:teuthology.orchestra.console.smithi120:power off output: Chassis Power Control: Down/Off 2024-03-28T16:06:53.448 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:53.471 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-28T16:06:57.399 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:57.439 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:57.472 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:06:57.509 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-28T16:06:57.548 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-28T16:06:57.582 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is on 2024-03-28T16:07:01.514 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:07:01.549 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:07:01.583 DEBUG:teuthology.orchestra.console.smithi120:pexpect command: ipmitool -H smithi120.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:07:01.624 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is on 2024-03-28T16:07:01.658 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-28T16:07:01.692 DEBUG:teuthology.orchestra.console.smithi120:check power output: Chassis Power is off 2024-03-28T16:07:01.793 INFO:teuthology.orchestra.console.smithi120:Power off completed 2024-03-28T16:07:05.629 DEBUG:teuthology.orchestra.console.smithi023:pexpect command: ipmitool -H smithi023.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:07:05.659 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-28T16:07:05.739 DEBUG:teuthology.orchestra.console.smithi023:check power output: Chassis Power is off 2024-03-28T16:07:05.769 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-28T16:07:05.841 INFO:teuthology.orchestra.console.smithi023:Power off completed 2024-03-28T16:07:05.870 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-28T16:07:05.976 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} duration: 1940.9261589050293 failure_reason: '"2024-03-28T15:52:37.274994+0000 mon.a (mon.0) 1144 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: mchangir success: false 2024-03-28T16:07:05.976 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T16:07:06.087 INFO:teuthology.run:FAIL